[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251215 08:57:21.169373 20109 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.163.126:44433
I20251215 08:57:21.171000 20109 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 08:57:21.172127 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:21.189584 20116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:21.189569 20115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:21.191159 20109 server_base.cc:1047] running on GCE node
W20251215 08:57:21.189663 20118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:22.403101 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:22.403369 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:22.403578 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789042403550 us; error 0 us; skew 500 ppm
I20251215 08:57:22.404429 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:22.411172 20109 webserver.cc:492] Webserver started at http://127.19.163.126:33859/ using document root <none> and password file <none>
I20251215 08:57:22.412114 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:22.412344 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:22.412864 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:22.418979 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/master-0-root/instance:
uuid: "8262533d139f4b09abad3556fe178493"
format_stamp: "Formatted at 2025-12-15 08:57:22 on dist-test-slave-s29j"
I20251215 08:57:22.427374 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20251215 08:57:22.432703 20124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:22.433861 20109 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251215 08:57:22.434219 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/master-0-root
uuid: "8262533d139f4b09abad3556fe178493"
format_stamp: "Formatted at 2025-12-15 08:57:22 on dist-test-slave-s29j"
I20251215 08:57:22.434588 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:22.489087 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:22.490731 20109 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 08:57:22.491197 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:22.566444 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.126:44433
I20251215 08:57:22.566532 20175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.126:44433 every 8 connection(s)
I20251215 08:57:22.574641 20176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:22.597219 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493: Bootstrap starting.
I20251215 08:57:22.603228 20176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:22.605193 20176 log.cc:826] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493: Log is configured to *not* fsync() on all Append() calls
I20251215 08:57:22.610090 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493: No bootstrap required, opened a new log
I20251215 08:57:22.630793 20176 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262533d139f4b09abad3556fe178493" member_type: VOTER }
I20251215 08:57:22.631376 20176 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:22.631641 20176 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8262533d139f4b09abad3556fe178493, State: Initialized, Role: FOLLOWER
I20251215 08:57:22.632516 20176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262533d139f4b09abad3556fe178493" member_type: VOTER }
I20251215 08:57:22.633033 20176 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 08:57:22.633296 20176 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 08:57:22.633651 20176 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:22.637691 20176 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262533d139f4b09abad3556fe178493" member_type: VOTER }
I20251215 08:57:22.638367 20176 leader_election.cc:304] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8262533d139f4b09abad3556fe178493; no voters:
I20251215 08:57:22.640187 20176 leader_election.cc:290] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 08:57:22.640488 20179 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:22.642751 20179 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 1 LEADER]: Becoming Leader. State: Replica: 8262533d139f4b09abad3556fe178493, State: Running, Role: LEADER
I20251215 08:57:22.643715 20179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262533d139f4b09abad3556fe178493" member_type: VOTER }
I20251215 08:57:22.644855 20176 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 08:57:22.654358 20181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8262533d139f4b09abad3556fe178493. Latest consensus state: current_term: 1 leader_uuid: "8262533d139f4b09abad3556fe178493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262533d139f4b09abad3556fe178493" member_type: VOTER } }
I20251215 08:57:22.654229 20180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8262533d139f4b09abad3556fe178493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262533d139f4b09abad3556fe178493" member_type: VOTER } }
I20251215 08:57:22.654973 20181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [sys.catalog]: This master's current role is: LEADER
I20251215 08:57:22.655040 20180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [sys.catalog]: This master's current role is: LEADER
I20251215 08:57:22.658069 20188 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 08:57:22.671087 20188 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 08:57:22.672969 20109 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 08:57:22.687440 20188 catalog_manager.cc:1357] Generated new cluster ID: fd861509fd7c4166a1b7a192bee7bc63
I20251215 08:57:22.687744 20188 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 08:57:22.703752 20188 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 08:57:22.705349 20188 catalog_manager.cc:1514] Loading token signing keys...
I20251215 08:57:22.718048 20188 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493: Generated new TSK 0
I20251215 08:57:22.719007 20188 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 08:57:22.739708 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:22.748441 20198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:22.750368 20199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:22.754333 20201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:22.755093 20109 server_base.cc:1047] running on GCE node
I20251215 08:57:22.756008 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:22.756270 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:22.756476 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789042756453 us; error 0 us; skew 500 ppm
I20251215 08:57:22.757319 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:22.761530 20109 webserver.cc:492] Webserver started at http://127.19.163.65:32853/ using document root <none> and password file <none>
I20251215 08:57:22.762243 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:22.762490 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:22.762857 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:22.764390 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-0-root/instance:
uuid: "7094b3ef76d943deb6fb0e997921ee2d"
format_stamp: "Formatted at 2025-12-15 08:57:22 on dist-test-slave-s29j"
I20251215 08:57:22.771314 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20251215 08:57:22.776343 20206 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:22.777168 20109 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251215 08:57:22.777451 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-0-root
uuid: "7094b3ef76d943deb6fb0e997921ee2d"
format_stamp: "Formatted at 2025-12-15 08:57:22 on dist-test-slave-s29j"
I20251215 08:57:22.777736 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:22.792472 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:22.793809 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:22.796065 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:57:22.800587 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:57:22.800824 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:22.801096 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:57:22.801254 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:22.898198 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.65:44543
I20251215 08:57:22.898326 20268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.65:44543 every 8 connection(s)
I20251215 08:57:22.905267 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:22.916965 20273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:22.920068 20274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:22.928224 20109 server_base.cc:1047] running on GCE node
W20251215 08:57:22.929899 20276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:22.930871 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:22.931201 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:22.931463 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789042931444 us; error 0 us; skew 500 ppm
I20251215 08:57:22.932049 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:22.935096 20109 webserver.cc:492] Webserver started at http://127.19.163.66:38781/ using document root <none> and password file <none>
I20251215 08:57:22.935683 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:22.935935 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:22.936270 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:22.937628 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-1-root/instance:
uuid: "65b535a7151242c7b67512ae19f24375"
format_stamp: "Formatted at 2025-12-15 08:57:22 on dist-test-slave-s29j"
I20251215 08:57:22.939399 20269 heartbeater.cc:344] Connected to a master server at 127.19.163.126:44433
I20251215 08:57:22.939905 20269 heartbeater.cc:461] Registering TS with master...
I20251215 08:57:22.941114 20269 heartbeater.cc:507] Master 127.19.163.126:44433 requested a full tablet report, sending...
I20251215 08:57:22.944314 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20251215 08:57:22.944701 20141 ts_manager.cc:194] Registered new tserver with Master: 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543)
I20251215 08:57:22.946971 20141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44066
I20251215 08:57:22.949867 20281 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:22.950865 20109 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20251215 08:57:22.951195 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-1-root
uuid: "65b535a7151242c7b67512ae19f24375"
format_stamp: "Formatted at 2025-12-15 08:57:22 on dist-test-slave-s29j"
I20251215 08:57:22.951592 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:22.966986 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:22.968701 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:22.971272 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:57:22.975091 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:57:22.975335 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:22.975643 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:57:22.975847 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:23.064450 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.66:45691
I20251215 08:57:23.064522 20343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.66:45691 every 8 connection(s)
I20251215 08:57:23.072952 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:23.081223 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:23.083353 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:23.088356 20344 heartbeater.cc:344] Connected to a master server at 127.19.163.126:44433
I20251215 08:57:23.088893 20344 heartbeater.cc:461] Registering TS with master...
I20251215 08:57:23.090155 20344 heartbeater.cc:507] Master 127.19.163.126:44433 requested a full tablet report, sending...
I20251215 08:57:23.092895 20141 ts_manager.cc:194] Registered new tserver with Master: 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
W20251215 08:57:23.093587 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:23.094653 20109 server_base.cc:1047] running on GCE node
I20251215 08:57:23.094798 20141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44070
I20251215 08:57:23.095546 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:23.095796 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:23.096015 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789043095996 us; error 0 us; skew 500 ppm
I20251215 08:57:23.096771 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:23.099460 20109 webserver.cc:492] Webserver started at http://127.19.163.67:45845/ using document root <none> and password file <none>
I20251215 08:57:23.099941 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:23.100103 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:23.100337 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:23.101500 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-2-root/instance:
uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
format_stamp: "Formatted at 2025-12-15 08:57:23 on dist-test-slave-s29j"
I20251215 08:57:23.106078 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251215 08:57:23.109512 20355 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:23.110311 20109 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251215 08:57:23.110574 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-2-root
uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
format_stamp: "Formatted at 2025-12-15 08:57:23 on dist-test-slave-s29j"
I20251215 08:57:23.110852 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765789040996898-20109-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:23.142164 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:23.143426 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:23.145555 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:57:23.149225 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:57:23.149431 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:23.149657 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:57:23.149806 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:23.240085 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.67:36505
I20251215 08:57:23.240696 20417 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.67:36505 every 8 connection(s)
I20251215 08:57:23.256585 20418 heartbeater.cc:344] Connected to a master server at 127.19.163.126:44433
I20251215 08:57:23.257043 20418 heartbeater.cc:461] Registering TS with master...
I20251215 08:57:23.257972 20418 heartbeater.cc:507] Master 127.19.163.126:44433 requested a full tablet report, sending...
I20251215 08:57:23.260159 20141 ts_manager.cc:194] Registered new tserver with Master: 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:23.260916 20109 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014632305s
I20251215 08:57:23.261912 20141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44082
I20251215 08:57:23.264333 20109 test_util.cc:276] Using random seed: -861140176
I20251215 08:57:23.310614 20141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44090:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 08:57:23.314288 20141 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 08:57:23.418768 20234 tablet_service.cc:1505] Processing CreateTablet for tablet fdd0fca4ba614b3091bb89126c86e585 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251215 08:57:23.420105 20234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd0fca4ba614b3091bb89126c86e585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.421885 20233 tablet_service.cc:1505] Processing CreateTablet for tablet 371033e1f37b435cb195960318735bdb (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251215 08:57:23.422230 20232 tablet_service.cc:1505] Processing CreateTablet for tablet d6eb6ebde2ed48a2a28e6ca4fb98e9ed (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251215 08:57:23.422907 20233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371033e1f37b435cb195960318735bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.423554 20232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6eb6ebde2ed48a2a28e6ca4fb98e9ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.429317 20231 tablet_service.cc:1505] Processing CreateTablet for tablet 14b54d338b7a4703b79365781fbd008b (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251215 08:57:23.430678 20231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b54d338b7a4703b79365781fbd008b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.447342 20383 tablet_service.cc:1505] Processing CreateTablet for tablet 371033e1f37b435cb195960318735bdb (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251215 08:57:23.448212 20381 tablet_service.cc:1505] Processing CreateTablet for tablet d6eb6ebde2ed48a2a28e6ca4fb98e9ed (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251215 08:57:23.452342 20382 tablet_service.cc:1505] Processing CreateTablet for tablet fdd0fca4ba614b3091bb89126c86e585 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251215 08:57:23.453320 20380 tablet_service.cc:1505] Processing CreateTablet for tablet 14b54d338b7a4703b79365781fbd008b (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251215 08:57:23.454147 20383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371033e1f37b435cb195960318735bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.457480 20381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6eb6ebde2ed48a2a28e6ca4fb98e9ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.458570 20380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b54d338b7a4703b79365781fbd008b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.471258 20308 tablet_service.cc:1505] Processing CreateTablet for tablet 371033e1f37b435cb195960318735bdb (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251215 08:57:23.475500 20307 tablet_service.cc:1505] Processing CreateTablet for tablet d6eb6ebde2ed48a2a28e6ca4fb98e9ed (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251215 08:57:23.471763 20382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd0fca4ba614b3091bb89126c86e585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.479071 20308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371033e1f37b435cb195960318735bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.477957 20309 tablet_service.cc:1505] Processing CreateTablet for tablet fdd0fca4ba614b3091bb89126c86e585 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251215 08:57:23.486157 20307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6eb6ebde2ed48a2a28e6ca4fb98e9ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.497485 20309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd0fca4ba614b3091bb89126c86e585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.499891 20230 tablet_service.cc:1505] Processing CreateTablet for tablet cc0a711ba0be47e5ae07f94c30d64b72 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251215 08:57:23.501747 20230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc0a711ba0be47e5ae07f94c30d64b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.514092 20306 tablet_service.cc:1505] Processing CreateTablet for tablet 14b54d338b7a4703b79365781fbd008b (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251215 08:57:23.518322 20306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b54d338b7a4703b79365781fbd008b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.520644 20437 tablet_bootstrap.cc:492] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
I20251215 08:57:23.531757 20305 tablet_service.cc:1505] Processing CreateTablet for tablet cc0a711ba0be47e5ae07f94c30d64b72 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251215 08:57:23.534307 20305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc0a711ba0be47e5ae07f94c30d64b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.548730 20436 tablet_bootstrap.cc:492] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.554257 20381 tablet_service.cc:1505] Processing CreateTablet for tablet cc0a711ba0be47e5ae07f94c30d64b72 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251215 08:57:23.556202 20381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc0a711ba0be47e5ae07f94c30d64b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.573681 20436 tablet_bootstrap.cc:654] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.578316 20234 tablet_service.cc:1505] Processing CreateTablet for tablet 6d2a63a43b2c4a4a80f94a4083c343b1 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251215 08:57:23.579484 20234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d2a63a43b2c4a4a80f94a4083c343b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.573681 20437 tablet_bootstrap.cc:654] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.588919 20304 tablet_service.cc:1505] Processing CreateTablet for tablet 6d2a63a43b2c4a4a80f94a4083c343b1 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251215 08:57:23.594323 20304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d2a63a43b2c4a4a80f94a4083c343b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.597867 20380 tablet_service.cc:1505] Processing CreateTablet for tablet 6d2a63a43b2c4a4a80f94a4083c343b1 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251215 08:57:23.600525 20380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d2a63a43b2c4a4a80f94a4083c343b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.626379 20230 tablet_service.cc:1505] Processing CreateTablet for tablet 954f43ae181e4c239a500a79d517d2b1 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251215 08:57:23.628062 20230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954f43ae181e4c239a500a79d517d2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.630154 20305 tablet_service.cc:1505] Processing CreateTablet for tablet 954f43ae181e4c239a500a79d517d2b1 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251215 08:57:23.632037 20305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954f43ae181e4c239a500a79d517d2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.633242 20232 tablet_service.cc:1505] Processing CreateTablet for tablet 40fa4688edda40508124f1077bdc390f (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251215 08:57:23.634470 20232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40fa4688edda40508124f1077bdc390f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.635670 20383 tablet_service.cc:1505] Processing CreateTablet for tablet 40fa4688edda40508124f1077bdc390f (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251215 08:57:23.636896 20383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40fa4688edda40508124f1077bdc390f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.641986 20307 tablet_service.cc:1505] Processing CreateTablet for tablet 40fa4688edda40508124f1077bdc390f (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251215 08:57:23.642565 20436 tablet_bootstrap.cc:492] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.641177 20382 tablet_service.cc:1505] Processing CreateTablet for tablet 954f43ae181e4c239a500a79d517d2b1 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251215 08:57:23.644362 20439 tablet_bootstrap.cc:492] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:23.645279 20382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954f43ae181e4c239a500a79d517d2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.649570 20231 tablet_service.cc:1505] Processing CreateTablet for tablet 3000909d86d64aa8a3b265ce0fb0a913 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251215 08:57:23.650751 20231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3000909d86d64aa8a3b265ce0fb0a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.657660 20309 tablet_service.cc:1505] Processing CreateTablet for tablet 3000909d86d64aa8a3b265ce0fb0a913 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251215 08:57:23.643070 20436 ts_tablet_manager.cc:1403] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.130s user 0.013s sys 0.005s
I20251215 08:57:23.643185 20307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40fa4688edda40508124f1077bdc390f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.660154 20309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3000909d86d64aa8a3b265ce0fb0a913. 1 dirs total, 0 dirs full, 0 dirs failed
W20251215 08:57:23.662024 20189 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:57:23.666675 20437 tablet_bootstrap.cc:492] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:23.667137 20437 ts_tablet_manager.cc:1403] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.148s user 0.009s sys 0.022s
I20251215 08:57:23.671851 20380 tablet_service.cc:1505] Processing CreateTablet for tablet 3000909d86d64aa8a3b265ce0fb0a913 (DEFAULT_TABLE table=test-workload [id=8fb7a01013c549fe8c38daae96069d83]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251215 08:57:23.673098 20380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3000909d86d64aa8a3b265ce0fb0a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:23.666066 20436 raft_consensus.cc:359] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.683753 20436 raft_consensus.cc:385] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.684113 20436 raft_consensus.cc:740] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.688355 20437 raft_consensus.cc:359] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.689168 20436 consensus_queue.cc:260] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.690081 20439 tablet_bootstrap.cc:654] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.689186 20437 raft_consensus.cc:385] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.694856 20437 raft_consensus.cc:740] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.695772 20437 consensus_queue.cc:260] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.697626 20269 heartbeater.cc:499] Master 127.19.163.126:44433 was elected leader, sending a full tablet report...
I20251215 08:57:23.698850 20436 ts_tablet_manager.cc:1434] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.038s user 0.008s sys 0.001s
I20251215 08:57:23.699266 20439 tablet_bootstrap.cc:492] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:23.699738 20439 ts_tablet_manager.cc:1403] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.010s
I20251215 08:57:23.699889 20436 tablet_bootstrap.cc:492] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.702703 20439 raft_consensus.cc:359] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:23.703572 20439 raft_consensus.cc:385] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.703959 20439 raft_consensus.cc:740] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.704730 20439 consensus_queue.cc:260] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:23.706766 20436 tablet_bootstrap.cc:654] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.716152 20418 heartbeater.cc:499] Master 127.19.163.126:44433 was elected leader, sending a full tablet report...
I20251215 08:57:23.720992 20439 ts_tablet_manager.cc:1434] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.021s user 0.011s sys 0.000s
I20251215 08:57:23.721177 20436 tablet_bootstrap.cc:492] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.721659 20436 ts_tablet_manager.cc:1403] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.002s
I20251215 08:57:23.721858 20344 heartbeater.cc:499] Master 127.19.163.126:44433 was elected leader, sending a full tablet report...
I20251215 08:57:23.724488 20436 raft_consensus.cc:359] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.725237 20436 raft_consensus.cc:385] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.725530 20436 raft_consensus.cc:740] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.726249 20436 consensus_queue.cc:260] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.727975 20439 tablet_bootstrap.cc:492] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:23.729306 20436 ts_tablet_manager.cc:1434] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20251215 08:57:23.730182 20436 tablet_bootstrap.cc:492] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.736881 20436 tablet_bootstrap.cc:654] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.738539 20437 ts_tablet_manager.cc:1434] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.071s user 0.007s sys 0.002s
I20251215 08:57:23.738968 20439 tablet_bootstrap.cc:654] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.739435 20437 tablet_bootstrap.cc:492] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
W20251215 08:57:23.747044 20419 tablet.cc:2378] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 08:57:23.751633 20437 tablet_bootstrap.cc:654] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.753669 20439 tablet_bootstrap.cc:492] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:23.754244 20439 ts_tablet_manager.cc:1403] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.002s
I20251215 08:57:23.755815 20436 tablet_bootstrap.cc:492] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.756294 20436 ts_tablet_manager.cc:1403] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.009s
I20251215 08:57:23.757915 20439 raft_consensus.cc:359] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.758735 20439 raft_consensus.cc:385] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.759093 20439 raft_consensus.cc:740] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.758980 20436 raft_consensus.cc:359] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.759699 20436 raft_consensus.cc:385] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.760007 20436 raft_consensus.cc:740] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.759910 20439 consensus_queue.cc:260] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.760738 20436 consensus_queue.cc:260] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.762791 20439 ts_tablet_manager.cc:1434] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251215 08:57:23.763613 20437 tablet_bootstrap.cc:492] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:23.763691 20439 tablet_bootstrap.cc:492] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:23.764124 20437 ts_tablet_manager.cc:1403] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.000s
I20251215 08:57:23.764175 20436 ts_tablet_manager.cc:1434] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251215 08:57:23.765192 20436 tablet_bootstrap.cc:492] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.766865 20437 raft_consensus.cc:359] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.767576 20437 raft_consensus.cc:385] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.767896 20437 raft_consensus.cc:740] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.768715 20437 consensus_queue.cc:260] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.771086 20439 tablet_bootstrap.cc:654] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.771287 20437 ts_tablet_manager.cc:1434] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.007s user 0.006s sys 0.003s
I20251215 08:57:23.772233 20437 tablet_bootstrap.cc:492] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
I20251215 08:57:23.783778 20436 tablet_bootstrap.cc:654] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.787600 20437 tablet_bootstrap.cc:654] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.792299 20437 tablet_bootstrap.cc:492] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:23.792311 20439 tablet_bootstrap.cc:492] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:23.792956 20439 ts_tablet_manager.cc:1403] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.019s
I20251215 08:57:23.792959 20437 ts_tablet_manager.cc:1403] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.004s
I20251215 08:57:23.795828 20439 raft_consensus.cc:359] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.795955 20437 raft_consensus.cc:359] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.796608 20439 raft_consensus.cc:385] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.796903 20437 raft_consensus.cc:385] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.796913 20439 raft_consensus.cc:740] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.797470 20437 raft_consensus.cc:740] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.797850 20439 consensus_queue.cc:260] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.798214 20437 consensus_queue.cc:260] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.800545 20439 ts_tablet_manager.cc:1434] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251215 08:57:23.801471 20439 tablet_bootstrap.cc:492] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:23.808182 20439 tablet_bootstrap.cc:654] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.811865 20436 tablet_bootstrap.cc:492] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.812335 20436 ts_tablet_manager.cc:1403] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.017s
I20251215 08:57:23.814356 20437 ts_tablet_manager.cc:1434] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20251215 08:57:23.815398 20437 tablet_bootstrap.cc:492] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
I20251215 08:57:23.815002 20436 raft_consensus.cc:359] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:23.815815 20436 raft_consensus.cc:385] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.816125 20436 raft_consensus.cc:740] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.816794 20436 consensus_queue.cc:260] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:23.819387 20436 ts_tablet_manager.cc:1434] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20251215 08:57:23.820207 20436 tablet_bootstrap.cc:492] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.828181 20436 tablet_bootstrap.cc:654] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.836227 20439 tablet_bootstrap.cc:492] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:23.836782 20439 ts_tablet_manager.cc:1403] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.017s
I20251215 08:57:23.839526 20437 tablet_bootstrap.cc:654] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.839725 20436 tablet_bootstrap.cc:492] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.839619 20439 raft_consensus.cc:359] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.840233 20436 ts_tablet_manager.cc:1403] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.010s
I20251215 08:57:23.840447 20439 raft_consensus.cc:385] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.840809 20439 raft_consensus.cc:740] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.841583 20439 consensus_queue.cc:260] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.843003 20436 raft_consensus.cc:359] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.843709 20436 raft_consensus.cc:385] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.843981 20436 raft_consensus.cc:740] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.844306 20439 ts_tablet_manager.cc:1434] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:57:23.844620 20436 consensus_queue.cc:260] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.845275 20439 tablet_bootstrap.cc:492] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:23.847339 20436 ts_tablet_manager.cc:1434] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251215 08:57:23.848310 20436 tablet_bootstrap.cc:492] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.852401 20439 tablet_bootstrap.cc:654] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.854439 20436 tablet_bootstrap.cc:654] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.866096 20436 tablet_bootstrap.cc:492] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.866560 20436 ts_tablet_manager.cc:1403] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20251215 08:57:23.867797 20444 raft_consensus.cc:493] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:23.869762 20444 raft_consensus.cc:515] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.872114 20436 raft_consensus.cc:359] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.872879 20436 raft_consensus.cc:385] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.873245 20436 raft_consensus.cc:740] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.873342 20444 leader_election.cc:290] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:23.874047 20436 consensus_queue.cc:260] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.875419 20439 tablet_bootstrap.cc:492] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:23.875876 20439 ts_tablet_manager.cc:1403] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.012s
I20251215 08:57:23.876910 20436 ts_tablet_manager.cc:1434] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20251215 08:57:23.877933 20436 tablet_bootstrap.cc:492] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.878703 20439 raft_consensus.cc:359] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.879393 20439 raft_consensus.cc:385] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.879702 20439 raft_consensus.cc:740] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.880379 20439 consensus_queue.cc:260] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.883067 20439 ts_tablet_manager.cc:1434] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251215 08:57:23.883996 20439 tablet_bootstrap.cc:492] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:23.885370 20436 tablet_bootstrap.cc:654] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.887893 20441 raft_consensus.cc:493] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:23.888353 20441 raft_consensus.cc:515] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.890940 20439 tablet_bootstrap.cc:654] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.890915 20441 leader_election.cc:290] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:23.898156 20441 raft_consensus.cc:493] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:23.898624 20441 raft_consensus.cc:515] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.900451 20441 leader_election.cc:290] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:23.903709 20436 tablet_bootstrap.cc:492] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.903927 20439 tablet_bootstrap.cc:492] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:23.904311 20436 ts_tablet_manager.cc:1403] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.013s
I20251215 08:57:23.905700 20439 ts_tablet_manager.cc:1403] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20251215 08:57:23.907301 20436 raft_consensus.cc:359] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.908156 20436 raft_consensus.cc:385] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.908479 20436 raft_consensus.cc:740] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.908823 20439 raft_consensus.cc:359] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.909507 20439 raft_consensus.cc:385] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.909806 20439 raft_consensus.cc:740] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.910619 20441 raft_consensus.cc:493] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:23.910431 20439 consensus_queue.cc:260] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.911062 20441 raft_consensus.cc:515] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.912855 20441 leader_election.cc:290] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:23.913434 20439 ts_tablet_manager.cc:1434] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251215 08:57:23.914585 20439 tablet_bootstrap.cc:492] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:23.915059 20436 consensus_queue.cc:260] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.917729 20436 ts_tablet_manager.cc:1434] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20251215 08:57:23.918737 20436 tablet_bootstrap.cc:492] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.921289 20439 tablet_bootstrap.cc:654] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.925045 20436 tablet_bootstrap.cc:654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.925614 20437 tablet_bootstrap.cc:492] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:23.926179 20437 ts_tablet_manager.cc:1403] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.111s user 0.006s sys 0.011s
I20251215 08:57:23.929316 20437 raft_consensus.cc:359] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.930250 20437 raft_consensus.cc:385] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.930612 20437 raft_consensus.cc:740] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.931449 20437 consensus_queue.cc:260] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.944418 20436 tablet_bootstrap.cc:492] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.944882 20436 ts_tablet_manager.cc:1403] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.011s
I20251215 08:57:23.947292 20436 raft_consensus.cc:359] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.947939 20436 raft_consensus.cc:385] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.948211 20436 raft_consensus.cc:740] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.948777 20436 consensus_queue.cc:260] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.949388 20437 ts_tablet_manager.cc:1434] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20251215 08:57:23.950404 20437 tablet_bootstrap.cc:492] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
I20251215 08:57:23.951103 20436 ts_tablet_manager.cc:1434] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251215 08:57:23.951937 20436 tablet_bootstrap.cc:492] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d: Bootstrap starting.
I20251215 08:57:23.957376 20436 tablet_bootstrap.cc:654] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.960227 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0a711ba0be47e5ae07f94c30d64b72" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:23.961236 20393 raft_consensus.cc:2468] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 0.
I20251215 08:57:23.963027 20285 leader_election.cc:304] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:23.964015 20444 raft_consensus.cc:2804] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:23.964439 20444 raft_consensus.cc:493] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:23.964838 20444 raft_consensus.cc:3060] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:23.968000 20436 tablet_bootstrap.cc:492] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d: No bootstrap required, opened a new log
I20251215 08:57:23.968415 20436 ts_tablet_manager.cc:1403] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20251215 08:57:23.970813 20436 raft_consensus.cc:359] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.971428 20436 raft_consensus.cc:385] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:23.971702 20436 raft_consensus.cc:740] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Initialized, Role: FOLLOWER
I20251215 08:57:23.972278 20436 consensus_queue.cc:260] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.972965 20444 raft_consensus.cc:515] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.974702 20436 ts_tablet_manager.cc:1434] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20251215 08:57:23.975188 20444 leader_election.cc:290] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:23.976224 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0a711ba0be47e5ae07f94c30d64b72" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:23.976780 20393 raft_consensus.cc:3060] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:23.981958 20437 tablet_bootstrap.cc:654] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:23.986303 20393 raft_consensus.cc:2468] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 1.
I20251215 08:57:23.987452 20285 leader_election.cc:304] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:23.988165 20441 raft_consensus.cc:493] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:23.988603 20441 raft_consensus.cc:515] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:23.990446 20441 leader_election.cc:290] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:23.991194 20444 raft_consensus.cc:2804] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:23.993242 20444 raft_consensus.cc:697] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:23.994098 20444 consensus_queue.cc:237] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.001348 20437 tablet_bootstrap.cc:492] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:24.001879 20437 ts_tablet_manager.cc:1403] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.001s
I20251215 08:57:24.012461 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0a711ba0be47e5ae07f94c30d64b72" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.014174 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0a711ba0be47e5ae07f94c30d64b72" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:24.014133 20244 raft_consensus.cc:2468] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 0.
I20251215 08:57:24.014815 20439 tablet_bootstrap.cc:492] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:24.020318 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.020617 20140 catalog_manager.cc:5654] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.022046 20318 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.022699 20318 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.023738 20317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.024432 20317 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.025300 20439 ts_tablet_manager.cc:1403] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.111s user 0.008s sys 0.023s
I20251215 08:57:24.025503 20316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.026224 20316 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.028601 20439 raft_consensus.cc:359] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.029168 20439 raft_consensus.cc:385] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.029448 20439 raft_consensus.cc:740] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
W20251215 08:57:24.029704 20209 leader_election.cc:343] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.030165 20439 consensus_queue.cc:260] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.030560 20209 leader_election.cc:304] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:24.031432 20441 raft_consensus.cc:2804] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.030776 20437 raft_consensus.cc:359] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:24.031800 20441 raft_consensus.cc:493] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.031924 20437 raft_consensus.cc:385] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.032182 20441 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.032274 20437 raft_consensus.cc:740] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:24.032697 20439 ts_tablet_manager.cc:1434] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251215 08:57:24.033217 20437 consensus_queue.cc:260] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:24.035897 20437 ts_tablet_manager.cc:1434] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20251215 08:57:24.038833 20392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.041252 20392 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.039331 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.039880 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.040177 20439 tablet_bootstrap.cc:492] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
I20251215 08:57:24.040230 20391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.043108 20390 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.043166 20391 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.046680 20437 tablet_bootstrap.cc:492] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
W20251215 08:57:24.045195 20210 leader_election.cc:343] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.048759 20210 leader_election.cc:304] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d; no voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:24.051065 20209 leader_election.cc:304] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:24.054313 20437 tablet_bootstrap.cc:654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:24.052229 20441 raft_consensus.cc:515] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.055311 20439 tablet_bootstrap.cc:654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:24.056928 20441 leader_election.cc:290] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Requested vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.059125 20471 raft_consensus.cc:2749] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:24.060204 20391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.060181 20210 leader_election.cc:304] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.061376 20391 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.061439 20441 raft_consensus.cc:2804] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.062731 20441 raft_consensus.cc:493] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.063102 20441 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.066962 20437 tablet_bootstrap.cc:492] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:24.067435 20437 ts_tablet_manager.cc:1403] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.003s
I20251215 08:57:24.070035 20391 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.070632 20471 raft_consensus.cc:2804] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.072292 20471 raft_consensus.cc:493] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.072762 20471 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.071014 20316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:24.072957 20444 raft_consensus.cc:493] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.078514 20316 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.078670 20444 raft_consensus.cc:515] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.080740 20444 leader_election.cc:290] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.070398 20437 raft_consensus.cc:359] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.081250 20437 raft_consensus.cc:385] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.081528 20437 raft_consensus.cc:740] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:24.072636 20210 leader_election.cc:304] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.082607 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40fa4688edda40508124f1077bdc390f" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.083889 20244 raft_consensus.cc:2468] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 0.
I20251215 08:57:24.086287 20282 leader_election.cc:304] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:24.086352 20472 raft_consensus.cc:2804] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.086854 20391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40fa4688edda40508124f1077bdc390f" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.088245 20444 raft_consensus.cc:2804] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.090380 20444 raft_consensus.cc:493] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.091051 20444 raft_consensus.cc:3060] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:57:24.090991 20285 leader_election.cc:343] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.092993 20441 raft_consensus.cc:515] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.094051 20316 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.090766 20437 consensus_queue.cc:260] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.116492 20468 raft_consensus.cc:493] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.120978 20439 tablet_bootstrap.cc:492] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:24.121590 20439 ts_tablet_manager.cc:1403] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.082s user 0.020s sys 0.026s
I20251215 08:57:24.117779 20468 raft_consensus.cc:515] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.126932 20475 raft_consensus.cc:493] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.127436 20475 raft_consensus.cc:515] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:24.150681 20472 raft_consensus.cc:697] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:24.151691 20472 consensus_queue.cc:237] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.154999 20441 leader_election.cc:290] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Requested vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.156076 20475 leader_election.cc:290] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.156426 20391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.157070 20391 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.157505 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.158107 20390 raft_consensus.cc:2468] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.158565 20444 raft_consensus.cc:515] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.159271 20210 leader_election.cc:304] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.160673 20439 raft_consensus.cc:359] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.168892 20439 raft_consensus.cc:385] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.169122 20444 leader_election.cc:290] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.169374 20439 raft_consensus.cc:740] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:24.161693 20471 raft_consensus.cc:515] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.170086 20439 consensus_queue.cc:260] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.162735 20468 leader_election.cc:290] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.171315 20475 raft_consensus.cc:2804] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.172912 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40fa4688edda40508124f1077bdc390f" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.173269 20437 ts_tablet_manager.cc:1434] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.105s user 0.010s sys 0.001s
I20251215 08:57:24.173183 20475 raft_consensus.cc:493] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.166545 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.173745 20475 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.174257 20437 tablet_bootstrap.cc:492] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
I20251215 08:57:24.172814 20439 ts_tablet_manager.cc:1434] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.051s user 0.009s sys 0.008s
W20251215 08:57:24.174823 20285 leader_election.cc:343] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.175187 20439 tablet_bootstrap.cc:492] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375: Bootstrap starting.
W20251215 08:57:24.175437 20285 leader_election.cc:343] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.182698 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.183298 20390 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.177639 20444 raft_consensus.cc:493] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.181208 20471 leader_election.cc:290] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Requested vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.193351 20391 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.195071 20210 leader_election.cc:304] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.198793 20471 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.199085 20441 raft_consensus.cc:2804] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.199241 20471 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.210378 20390 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.213243 20210 leader_election.cc:304] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.214030 20472 raft_consensus.cc:2804] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.214100 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.214648 20472 raft_consensus.cc:697] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:24.187393 20444 raft_consensus.cc:515] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:24.211751 20441 raft_consensus.cc:697] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:24.238956 20439 tablet_bootstrap.cc:654] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:24.238888 20472 consensus_queue.cc:237] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.242144 20437 tablet_bootstrap.cc:654] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:24.241765 20139 catalog_manager.cc:5654] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 1 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.244416 20441 consensus_queue.cc:237] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.245100 20244 raft_consensus.cc:2468] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 0.
I20251215 08:57:24.247681 20443 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.248274 20443 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.251266 20443 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.256587 20472 raft_consensus.cc:493] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.257218 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40fa4688edda40508124f1077bdc390f" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:24.257144 20472 raft_consensus.cc:515] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.257797 20244 raft_consensus.cc:3060] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.259009 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.259164 20472 leader_election.cc:290] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.259630 20390 raft_consensus.cc:2468] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 0.
I20251215 08:57:24.260818 20285 leader_election.cc:304] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.261283 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3000909d86d64aa8a3b265ce0fb0a913" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.261816 20479 raft_consensus.cc:2804] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.262207 20479 raft_consensus.cc:493] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.262542 20479 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:57:24.262599 20210 leader_election.cc:343] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.263302 20317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.264636 20318 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:24.265211 20318 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.265321 20316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:24.265862 20316 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.272189 20318 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.273332 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.273659 20318 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.273980 20243 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 0.
I20251215 08:57:24.274266 20318 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 0.
I20251215 08:57:24.274266 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3000909d86d64aa8a3b265ce0fb0a913" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.275450 20356 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.276283 20443 raft_consensus.cc:2804] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.276387 20316 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.276679 20443 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.276793 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.277117 20443 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.278673 20209 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:24.279541 20472 raft_consensus.cc:2804] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.279909 20472 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.280128 20472 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.277091 20444 leader_election.cc:290] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.282418 20244 raft_consensus.cc:2468] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 1.
I20251215 08:57:24.282255 20479 raft_consensus.cc:515] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
W20251215 08:57:24.283099 20209 leader_election.cc:343] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.284960 20209 leader_election.cc:304] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d; no voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:24.283844 20282 leader_election.cc:304] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters: 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:24.286125 20478 raft_consensus.cc:2749] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:24.286383 20472 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.287668 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" is_pre_election: true
I20251215 08:57:24.288300 20471 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.288641 20317 raft_consensus.cc:2393] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7094b3ef76d943deb6fb0e997921ee2d in current term 1: Already voted for candidate 65b535a7151242c7b67512ae19f24375 in this term.
I20251215 08:57:24.289032 20472 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Requested vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.289085 20316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.289269 20391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.289881 20316 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 0.
I20251215 08:57:24.291659 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:24.292186 20319 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.292590 20475 raft_consensus.cc:515] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:24.293262 20444 raft_consensus.cc:2804] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.293699 20444 raft_consensus.cc:493] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.293823 20243 raft_consensus.cc:2393] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b535a7151242c7b67512ae19f24375 in current term 1: Already voted for candidate 7094b3ef76d943deb6fb0e997921ee2d in this term.
I20251215 08:57:24.294068 20444 raft_consensus.cc:3060] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.295223 20475 leader_election.cc:290] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Requested vote from peers 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.295688 20391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.296252 20391 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.298261 20319 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.285611 20443 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.301143 20479 leader_election.cc:290] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.302253 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.303236 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:24.303467 20444 raft_consensus.cc:515] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.303983 20243 raft_consensus.cc:2393] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 65b535a7151242c7b67512ae19f24375 in current term 1: Already voted for candidate 7094b3ef76d943deb6fb0e997921ee2d in this term.
I20251215 08:57:24.303632 20209 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters: 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:24.304764 20439 tablet_bootstrap.cc:492] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375: No bootstrap required, opened a new log
I20251215 08:57:24.305066 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:24.305243 20439 ts_tablet_manager.cc:1403] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375: Time spent bootstrapping tablet: real 0.130s user 0.009s sys 0.011s
I20251215 08:57:24.306874 20282 leader_election.cc:304] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b535a7151242c7b67512ae19f24375; no voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:24.308063 20282 leader_election.cc:304] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters: 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:24.309027 20468 raft_consensus.cc:2804] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.309171 20439 raft_consensus.cc:359] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.310166 20439 raft_consensus.cc:385] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.310503 20439 raft_consensus.cc:740] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Initialized, Role: FOLLOWER
I20251215 08:57:24.311127 20444 leader_election.cc:290] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:24.311283 20439 consensus_queue.cc:260] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.312734 20390 raft_consensus.cc:2393] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7094b3ef76d943deb6fb0e997921ee2d in current term 1: Already voted for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in this term.
I20251215 08:57:24.312606 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:24.313606 20319 raft_consensus.cc:2393] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7094b3ef76d943deb6fb0e997921ee2d in current term 1: Already voted for candidate 65b535a7151242c7b67512ae19f24375 in this term.
I20251215 08:57:24.313773 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:24.314291 20244 raft_consensus.cc:3060] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.314510 20479 raft_consensus.cc:2749] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251215 08:57:24.315248 20285 leader_election.cc:343] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:24.306252 20475 raft_consensus.cc:2804] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.316329 20475 raft_consensus.cc:697] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:24.317348 20391 raft_consensus.cc:2468] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 1.
I20251215 08:57:24.317432 20475 consensus_queue.cc:237] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.318918 20210 leader_election.cc:304] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.320271 20471 raft_consensus.cc:2804] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.320806 20471 raft_consensus.cc:697] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:24.321889 20471 consensus_queue.cc:237] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:24.329303 20443 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.332711 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:24.333407 20244 raft_consensus.cc:2468] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 1.
I20251215 08:57:24.335813 20282 leader_election.cc:304] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters: 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:24.337203 20444 raft_consensus.cc:2804] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.337239 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:24.337831 20444 raft_consensus.cc:697] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:24.338434 20319 raft_consensus.cc:2393] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in current term 1: Already voted for candidate 7094b3ef76d943deb6fb0e997921ee2d in this term.
I20251215 08:57:24.338968 20444 consensus_queue.cc:237] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.340397 20468 raft_consensus.cc:697] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:24.341404 20439 ts_tablet_manager.cc:1434] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20251215 08:57:24.341486 20358 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 829fd63ab8fc4c0ab6380914a02dc2d6; no voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d
I20251215 08:57:24.342487 20443 raft_consensus.cc:2749] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251215 08:57:24.341305 20468 consensus_queue.cc:237] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.344380 20141 catalog_manager.cc:5654] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 1 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.347586 20141 catalog_manager.cc:5654] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 1 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.351948 20437 tablet_bootstrap.cc:492] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:24.352461 20437 ts_tablet_manager.cc:1403] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.178s user 0.020s sys 0.000s
I20251215 08:57:24.355422 20437 raft_consensus.cc:359] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.356112 20437 raft_consensus.cc:385] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.356384 20437 raft_consensus.cc:740] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:24.357072 20437 consensus_queue.cc:260] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.359426 20437 ts_tablet_manager.cc:1434] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20251215 08:57:24.360359 20437 tablet_bootstrap.cc:492] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
I20251215 08:57:24.363421 20140 catalog_manager.cc:5654] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.366441 20437 tablet_bootstrap.cc:654] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:24.375523 20437 tablet_bootstrap.cc:492] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:24.375945 20437 ts_tablet_manager.cc:1403] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20251215 08:57:24.378234 20437 raft_consensus.cc:359] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.378904 20437 raft_consensus.cc:385] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.379185 20437 raft_consensus.cc:740] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:24.375937 20139 catalog_manager.cc:5654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 1 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.379771 20437 consensus_queue.cc:260] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.381863 20437 ts_tablet_manager.cc:1434] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251215 08:57:24.382680 20437 tablet_bootstrap.cc:492] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Bootstrap starting.
I20251215 08:57:24.385572 20138 catalog_manager.cc:5654] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.386691 20139 catalog_manager.cc:5654] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 1 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:24.390097 20437 tablet_bootstrap.cc:654] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:24.397866 20437 tablet_bootstrap.cc:492] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: No bootstrap required, opened a new log
I20251215 08:57:24.398437 20437 ts_tablet_manager.cc:1403] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20251215 08:57:24.403823 20437 raft_consensus.cc:359] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.404702 20437 raft_consensus.cc:385] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:24.405059 20437 raft_consensus.cc:740] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Initialized, Role: FOLLOWER
I20251215 08:57:24.405686 20437 consensus_queue.cc:260] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.408731 20437 ts_tablet_manager.cc:1434] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20251215 08:57:24.436141 20443 raft_consensus.cc:493] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.436648 20443 raft_consensus.cc:515] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.438007 20468 consensus_queue.cc:1048] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.438506 20443 leader_election.cc:290] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.439330 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.439945 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.440347 20244 raft_consensus.cc:2393] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in current term 1: Already voted for candidate 65b535a7151242c7b67512ae19f24375 in this term.
I20251215 08:57:24.441783 20356 leader_election.cc:304] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 829fd63ab8fc4c0ab6380914a02dc2d6; no voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d
I20251215 08:57:24.442586 20443 raft_consensus.cc:3060] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.451392 20244 raft_consensus.cc:3060] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.457769 20443 raft_consensus.cc:2749] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:24.461248 20468 consensus_queue.cc:1048] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.608635 20443 raft_consensus.cc:493] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.609163 20443 raft_consensus.cc:515] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.611124 20443 leader_election.cc:290] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.612193 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40fa4688edda40508124f1077bdc390f" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.612923 20244 raft_consensus.cc:2393] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in current term 1: Already voted for candidate 65b535a7151242c7b67512ae19f24375 in this term.
I20251215 08:57:24.612679 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40fa4688edda40508124f1077bdc390f" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.614169 20358 leader_election.cc:304] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 829fd63ab8fc4c0ab6380914a02dc2d6; no voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d
I20251215 08:57:24.614789 20443 raft_consensus.cc:3060] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.619392 20443 raft_consensus.cc:2749] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:24.668027 20443 raft_consensus.cc:493] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:24.668462 20443 raft_consensus.cc:515] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.670336 20443 leader_election.cc:290] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.671526 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3000909d86d64aa8a3b265ce0fb0a913" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375" is_pre_election: true
I20251215 08:57:24.672036 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3000909d86d64aa8a3b265ce0fb0a913" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d" is_pre_election: true
I20251215 08:57:24.672183 20319 raft_consensus.cc:2468] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 0.
I20251215 08:57:24.672703 20244 raft_consensus.cc:2468] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 0.
I20251215 08:57:24.674047 20356 leader_election.cc:304] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
W20251215 08:57:24.674422 20189 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:57:24.674827 20443 raft_consensus.cc:2804] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:24.675223 20443 raft_consensus.cc:493] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:24.675537 20443 raft_consensus.cc:3060] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.680070 20443 raft_consensus.cc:515] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.681634 20443 leader_election.cc:290] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:24.682410 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3000909d86d64aa8a3b265ce0fb0a913" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:24.682761 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3000909d86d64aa8a3b265ce0fb0a913" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:24.683000 20244 raft_consensus.cc:3060] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.683310 20319 raft_consensus.cc:3060] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:24.689620 20319 raft_consensus.cc:2468] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 1.
I20251215 08:57:24.689620 20244 raft_consensus.cc:2468] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 1.
I20251215 08:57:24.690923 20356 leader_election.cc:304] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:24.691637 20443 raft_consensus.cc:2804] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:24.692694 20443 raft_consensus.cc:697] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 LEADER]: Becoming Leader. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Running, Role: LEADER
I20251215 08:57:24.693456 20443 consensus_queue.cc:237] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:24.700289 20138 catalog_manager.cc:5654] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67). New cstate: current_term: 1 leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:24.721736 20471 consensus_queue.cc:1048] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.729046 20471 consensus_queue.cc:1048] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.735384 20468 consensus_queue.cc:1048] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.744581 20441 consensus_queue.cc:1048] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.750442 20471 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.756490 20471 consensus_queue.cc:1048] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:57:24.763705 20471 consensus_queue.cc:1048] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:57:24.788205 20475 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.790239 20479 consensus_queue.cc:1048] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.814973 20478 consensus_queue.cc:1048] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.825752 20444 consensus_queue.cc:1048] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:57:24.837821 20475 consensus_queue.cc:1048] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.842404 20510 consensus_queue.cc:1048] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:57:24.880852 20444 consensus_queue.cc:1048] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:24.945801 20244 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
mode: GRACEFUL
new_leader_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:49598
I20251215 08:57:24.946391 20244 raft_consensus.cc:606] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Received request to transfer leadership to TS 65b535a7151242c7b67512ae19f24375
I20251215 08:57:25.129652 20443 consensus_queue.cc:1048] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:25.139856 20443 consensus_queue.cc:1048] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:25.356431 20471 raft_consensus.cc:993] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d: : Instructing follower 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:25.356956 20510 raft_consensus.cc:1081] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Signalling peer 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:25.358841 20319 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:53154
I20251215 08:57:25.359450 20319 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:25.359894 20319 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:25.364487 20319 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:25.366168 20319 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:25.367180 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:25.367219 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:25.367861 20393 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:25.367957 20244 raft_consensus.cc:3055] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:25.368345 20244 raft_consensus.cc:740] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:25.369217 20244 consensus_queue.cc:260] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:25.370631 20244 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:25.372555 20393 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:25.373739 20285 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:25.374377 20479 raft_consensus.cc:2804] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:25.374754 20479 raft_consensus.cc:697] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:25.375459 20479 consensus_queue.cc:237] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:25.379992 20244 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:25.382371 20138 catalog_manager.cc:5654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 1 to 2, leader changed from 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65) to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:25.798586 20393 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:25.799926 20479 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:25.808290 20244 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:25.810310 20468 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:27.981535 20109 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20251215 08:57:27.982286 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:28.001307 20319 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
mode: GRACEFUL
new_leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:42924
I20251215 08:57:28.001873 20319 raft_consensus.cc:606] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Received request to transfer leadership to TS 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:28.011617 20244 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
mode: GRACEFUL
new_leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:49602
I20251215 08:57:28.012109 20244 raft_consensus.cc:606] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Received request to transfer leadership to TS 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:28.013293 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251215 08:57:28.013723 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:28.060076 20533 raft_consensus.cc:993] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: : Instructing follower 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:28.060542 20479 raft_consensus.cc:1081] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Signalling peer 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:28.061951 20393 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:44308
I20251215 08:57:28.062413 20393 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:28.062695 20393 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:28.067199 20393 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:28.068668 20393 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 3 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:28.069636 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:28.069782 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:28.070271 20244 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:28.070358 20319 raft_consensus.cc:3055] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:57:28.070713 20319 raft_consensus.cc:740] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:28.071352 20319 consensus_queue.cc:260] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:28.072484 20319 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:28.074867 20244 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 3.
I20251215 08:57:28.075906 20356 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:28.076700 20443 raft_consensus.cc:2804] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:57:28.077167 20319 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 3.
I20251215 08:57:28.077970 20443 raft_consensus.cc:697] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 LEADER]: Becoming Leader. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Running, Role: LEADER
I20251215 08:57:28.078832 20443 consensus_queue.cc:237] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:28.085809 20138 catalog_manager.cc:5654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 reported cstate change: term changed from 2 to 3, leader changed from 65b535a7151242c7b67512ae19f24375 (127.19.163.66) to 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67). New cstate: current_term: 3 leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:28.364341 20511 raft_consensus.cc:993] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d: : Instructing follower 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:28.364730 20511 raft_consensus.cc:1081] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Signalling peer 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:28.366104 20393 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed"
dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:44324
I20251215 08:57:28.366528 20393 raft_consensus.cc:493] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:28.366843 20393 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:28.371423 20393 raft_consensus.cc:515] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:28.372968 20393 leader_election.cc:290] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 2 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:28.373837 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:28.374033 20319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:28.374322 20244 raft_consensus.cc:3055] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:28.374507 20319 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:28.374650 20244 raft_consensus.cc:740] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:28.375370 20244 consensus_queue.cc:260] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:28.376250 20244 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:28.378907 20319 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 2.
I20251215 08:57:28.379837 20358 leader_election.cc:304] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:28.380488 20443 raft_consensus.cc:2804] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:28.380663 20244 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 2.
I20251215 08:57:28.380877 20443 raft_consensus.cc:697] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 LEADER]: Becoming Leader. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Running, Role: LEADER
I20251215 08:57:28.381736 20443 consensus_queue.cc:237] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:28.390820 20138 catalog_manager.cc:5654] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 reported cstate change: term changed from 1 to 2, leader changed from 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65) to 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67). New cstate: current_term: 2 leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:28.577466 20319 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:28.578725 20443 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:28.588550 20244 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:28.590118 20537 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:28.816606 20244 raft_consensus.cc:1275] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:28.817736 20537 consensus_queue.cc:1048] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:28.825879 20319 raft_consensus.cc:1275] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:28.827402 20443 consensus_queue.cc:1048] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:29.014631 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:29.019670 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:29.020044 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:30.020859 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:30.025810 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:30.026155 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:31.026993 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:31.031754 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:31.032114 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:32.033089 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:32.038560 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:32.038970 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:33.040077 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:33.045699 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:33.046170 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:34.047075 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:34.051951 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:34.052474 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:35.053475 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:35.058200 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:35.058655 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:36.059582 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:36.064618 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:36.065084 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:37.066068 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:37.071177 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:37.071604 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:38.100174 20109 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20251215 08:57:38.105278 20243 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
mode: GRACEFUL
new_leader_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:49598
I20251215 08:57:38.106016 20243 raft_consensus.cc:606] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Received request to transfer leadership to TS 65b535a7151242c7b67512ae19f24375
I20251215 08:57:38.111233 20243 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "371033e1f37b435cb195960318735bdb"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
mode: GRACEFUL
new_leader_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:49598
I20251215 08:57:38.111742 20243 raft_consensus.cc:606] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Received request to transfer leadership to TS 65b535a7151242c7b67512ae19f24375
I20251215 08:57:38.126940 20393 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed"
dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
mode: GRACEFUL
new_leader_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:56988
I20251215 08:57:38.127395 20393 raft_consensus.cc:606] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 LEADER]: Received request to transfer leadership to TS 65b535a7151242c7b67512ae19f24375
I20251215 08:57:38.132469 20243 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
mode: GRACEFUL
new_leader_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:49598
I20251215 08:57:38.133035 20243 raft_consensus.cc:606] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Received request to transfer leadership to TS 65b535a7151242c7b67512ae19f24375
I20251215 08:57:38.140741 20393 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
mode: GRACEFUL
new_leader_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:56988
I20251215 08:57:38.141247 20393 raft_consensus.cc:606] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 LEADER]: Received request to transfer leadership to TS 65b535a7151242c7b67512ae19f24375
I20251215 08:57:38.168061 20561 raft_consensus.cc:993] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d: : Instructing follower 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.168681 20559 raft_consensus.cc:1081] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Signalling peer 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.170265 20319 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:53154
I20251215 08:57:38.170747 20319 raft_consensus.cc:493] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:38.171079 20319 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.190135 20561 raft_consensus.cc:993] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d: : Instructing follower 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.190634 20559 raft_consensus.cc:1081] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Signalling peer 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.192222 20317 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:53154
I20251215 08:57:38.192756 20317 raft_consensus.cc:493] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:38.193090 20317 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.216158 20560 raft_consensus.cc:993] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: : Instructing follower 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.216589 20560 raft_consensus.cc:1081] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 LEADER]: Signalling peer 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.218127 20316 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:53156
I20251215 08:57:38.218624 20316 raft_consensus.cc:493] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:38.218919 20316 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:38.269969 20319 raft_consensus.cc:515] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:38.272749 20319 leader_election.cc:290] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:38.275079 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:38.275039 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:38.276022 20393 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.276124 20243 raft_consensus.cc:3055] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:38.276542 20243 raft_consensus.cc:740] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:38.277374 20243 consensus_queue.cc:260] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:38.278671 20243 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.365918 20316 raft_consensus.cc:515] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.365878 20393 raft_consensus.cc:2468] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:38.366349 20317 raft_consensus.cc:515] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.365878 20243 raft_consensus.cc:2468] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:38.369146 20282 leader_election.cc:304] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:38.370012 20244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:38.370635 20244 raft_consensus.cc:3055] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:38.370788 20479 raft_consensus.cc:2804] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:38.371129 20244 raft_consensus.cc:740] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:38.370496 20316 leader_election.cc:290] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 3 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:38.371263 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:38.371856 20243 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:38.370683 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:38.371918 20244 consensus_queue.cc:260] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.373024 20393 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.373553 20244 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.377707 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6eb6ebde2ed48a2a28e6ca4fb98e9ed" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:38.378414 20390 raft_consensus.cc:3055] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:57:38.378715 20390 raft_consensus.cc:740] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Running, Role: LEADER
I20251215 08:57:38.379479 20390 consensus_queue.cc:260] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.380944 20390 raft_consensus.cc:3060] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:38.381089 20479 raft_consensus.cc:697] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:38.369187 20317 leader_election.cc:290] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:38.382050 20479 consensus_queue.cc:237] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:38.391137 20138 catalog_manager.cc:5654] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 1 to 2, leader changed from 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65) to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:38.420131 20561 raft_consensus.cc:993] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d: : Instructing follower 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.420567 20559 raft_consensus.cc:1081] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Signalling peer 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.421988 20317 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "371033e1f37b435cb195960318735bdb"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:53154
I20251215 08:57:38.422468 20317 raft_consensus.cc:493] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:38.422770 20317 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.462343 20393 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:38.462707 20244 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:38.462381 20317 raft_consensus.cc:515] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.462350 20243 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 3.
I20251215 08:57:38.464860 20282 leader_election.cc:304] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:38.465271 20317 leader_election.cc:290] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:38.465754 20282 leader_election.cc:304] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:38.466084 20479 raft_consensus.cc:2804] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:38.466751 20565 raft_consensus.cc:2804] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:57:38.467720 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:38.468320 20243 raft_consensus.cc:3055] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:38.468604 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:38.468631 20243 raft_consensus.cc:740] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:38.469151 20393 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.469588 20243 consensus_queue.cc:260] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.470845 20243 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:38.471253 20565 raft_consensus.cc:697] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 3 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:38.472318 20565 consensus_queue.cc:237] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.475673 20479 raft_consensus.cc:697] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:38.476608 20479 consensus_queue.cc:237] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.482910 20138 catalog_manager.cc:5654] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 2 to 3, leader changed from 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67) to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 3 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:38.494056 20138 catalog_manager.cc:5654] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 1 to 2, leader changed from 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65) to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:38.506372 20564 raft_consensus.cc:993] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: : Instructing follower 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.506801 20564 raft_consensus.cc:1081] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 LEADER]: Signalling peer 65b535a7151242c7b67512ae19f24375 to start an election
I20251215 08:57:38.508183 20317 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
from {username='slave'} at 127.0.0.1:53156
I20251215 08:57:38.508637 20317 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:38.508894 20317 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:57:38.559003 20243 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:38.559346 20393 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 2.
I20251215 08:57:38.559002 20390 raft_consensus.cc:2468] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 3.
I20251215 08:57:38.560281 20317 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.561524 20285 leader_election.cc:304] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:38.562296 20317 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 4 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:38.563084 20479 raft_consensus.cc:2804] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:38.563227 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:38.563597 20479 raft_consensus.cc:697] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:38.563896 20243 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:57:38.564066 20390 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "65b535a7151242c7b67512ae19f24375" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:38.564677 20390 raft_consensus.cc:3055] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:57:38.564429 20479 consensus_queue.cc:237] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.564988 20390 raft_consensus.cc:740] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Running, Role: LEADER
I20251215 08:57:38.565790 20390 consensus_queue.cc:260] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.567137 20390 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:57:38.572719 20138 catalog_manager.cc:5654] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 1 to 2, leader changed from 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65) to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:38.656117 20243 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 4.
I20251215 08:57:38.656117 20390 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b535a7151242c7b67512ae19f24375 in term 4.
I20251215 08:57:38.657581 20282 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b535a7151242c7b67512ae19f24375, 7094b3ef76d943deb6fb0e997921ee2d; no voters:
I20251215 08:57:38.658228 20479 raft_consensus.cc:2804] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:57:38.658592 20479 raft_consensus.cc:697] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 LEADER]: Becoming Leader. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:38.659432 20479 consensus_queue.cc:237] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:38.665568 20138 catalog_manager.cc:5654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 reported cstate change: term changed from 3 to 4, leader changed from 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67) to 65b535a7151242c7b67512ae19f24375 (127.19.163.66). New cstate: current_term: 4 leader_uuid: "65b535a7151242c7b67512ae19f24375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:38.870427 20390 raft_consensus.cc:1275] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:38.871568 20479 consensus_queue.cc:1048] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:38.883647 20244 raft_consensus.cc:1275] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:38.885063 20565 consensus_queue.cc:1048] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:38.907919 20390 raft_consensus.cc:1275] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:38.909250 20565 consensus_queue.cc:1048] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:38.914160 20393 raft_consensus.cc:1275] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:38.915592 20565 consensus_queue.cc:1048] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:38.922453 20244 raft_consensus.cc:1275] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:38.925526 20479 consensus_queue.cc:1048] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:38.931767 20244 raft_consensus.cc:1275] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:38.936010 20565 consensus_queue.cc:1048] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:39.067903 20390 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 4 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:57:39.069159 20565 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:57:39.078086 20243 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 4 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:57:39.080214 20565 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:57:39.155373 20390 raft_consensus.cc:1275] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:39.156617 20572 consensus_queue.cc:1048] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:39.164992 20243 raft_consensus.cc:1275] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Refusing update from remote peer 65b535a7151242c7b67512ae19f24375: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:39.166558 20572 consensus_queue.cc:1048] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:57:41.175705 20109 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20251215 08:57:41.176366 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:41.183585 20317 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
mode: GRACEFUL
new_leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
from {username='slave'} at 127.0.0.1:42924
I20251215 08:57:41.184178 20317 raft_consensus.cc:606] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Received request to transfer leadership to TS 7094b3ef76d943deb6fb0e997921ee2d
I20251215 08:57:41.186408 20317 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "371033e1f37b435cb195960318735bdb"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
mode: GRACEFUL
new_leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:42924
I20251215 08:57:41.186964 20317 raft_consensus.cc:606] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Received request to transfer leadership to TS 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:41.188964 20317 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
mode: GRACEFUL
new_leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:42924
I20251215 08:57:41.189568 20317 raft_consensus.cc:606] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 LEADER]: Received request to transfer leadership to TS 829fd63ab8fc4c0ab6380914a02dc2d6
I20251215 08:57:41.191856 20317 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
mode: GRACEFUL
new_leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
from {username='slave'} at 127.0.0.1:42924
I20251215 08:57:41.192260 20317 raft_consensus.cc:606] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Received request to transfer leadership to TS 7094b3ef76d943deb6fb0e997921ee2d
I20251215 08:57:41.194056 20317 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585"
dest_uuid: "65b535a7151242c7b67512ae19f24375"
mode: GRACEFUL
new_leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
from {username='slave'} at 127.0.0.1:42924
I20251215 08:57:41.194451 20317 raft_consensus.cc:606] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Received request to transfer leadership to TS 7094b3ef76d943deb6fb0e997921ee2d
I20251215 08:57:41.195293 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:57:41.195806 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:41.289738 20590 raft_consensus.cc:993] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: : Instructing follower 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:41.290431 20479 raft_consensus.cc:1081] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 LEADER]: Signalling peer 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:41.292200 20393 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1"
dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:44308
I20251215 08:57:41.292986 20393 raft_consensus.cc:493] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:41.293385 20393 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:57:41.299994 20393 raft_consensus.cc:515] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.302327 20393 leader_election.cc:290] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 5 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:41.303959 20317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:41.304015 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d2a63a43b2c4a4a80f94a4083c343b1" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:41.304826 20317 raft_consensus.cc:3055] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:57:41.304984 20243 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:57:41.305184 20317 raft_consensus.cc:740] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:41.305928 20317 consensus_queue.cc:260] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.307118 20317 raft_consensus.cc:3060] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:57:41.309976 20243 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 5.
I20251215 08:57:41.311343 20356 leader_election.cc:304] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:41.312356 20560 raft_consensus.cc:2804] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:57:41.312960 20317 raft_consensus.cc:2468] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 5.
I20251215 08:57:41.313586 20560 raft_consensus.cc:697] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 5 LEADER]: Becoming Leader. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Running, Role: LEADER
I20251215 08:57:41.315625 20590 raft_consensus.cc:993] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375: : Instructing follower 7094b3ef76d943deb6fb0e997921ee2d to start an election
I20251215 08:57:41.315287 20560 consensus_queue.cc:237] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.316071 20479 raft_consensus.cc:1081] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Signalling peer 7094b3ef76d943deb6fb0e997921ee2d to start an election
I20251215 08:57:41.317524 20243 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
from {username='slave'} at 127.0.0.1:35162
I20251215 08:57:41.317962 20243 raft_consensus.cc:493] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:41.318208 20243 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.324455 20243 raft_consensus.cc:515] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.325865 20138 catalog_manager.cc:5654] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 reported cstate change: term changed from 4 to 5, leader changed from 65b535a7151242c7b67512ae19f24375 (127.19.163.66) to 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67). New cstate: current_term: 5 leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:41.327072 20243 leader_election.cc:290] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 3 election: Requested vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:41.328080 20317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:41.328330 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b54d338b7a4703b79365781fbd008b" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:41.328681 20317 raft_consensus.cc:3055] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:57:41.328933 20393 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.329072 20317 raft_consensus.cc:740] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:41.330145 20317 consensus_queue.cc:260] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.331646 20317 raft_consensus.cc:3060] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.337844 20393 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 3.
I20251215 08:57:41.339423 20210 leader_election.cc:304] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:41.340385 20317 raft_consensus.cc:2468] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 3.
I20251215 08:57:41.341534 20596 raft_consensus.cc:2804] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:57:41.342985 20596 raft_consensus.cc:697] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 3 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:41.344205 20596 consensus_queue.cc:237] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.352181 20138 catalog_manager.cc:5654] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 2 to 3, leader changed from 65b535a7151242c7b67512ae19f24375 (127.19.163.66) to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 3 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:41.356163 20590 raft_consensus.cc:993] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375: : Instructing follower 7094b3ef76d943deb6fb0e997921ee2d to start an election
I20251215 08:57:41.356684 20479 raft_consensus.cc:1081] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Signalling peer 7094b3ef76d943deb6fb0e997921ee2d to start an election
I20251215 08:57:41.358258 20243 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
from {username='slave'} at 127.0.0.1:35162
I20251215 08:57:41.358791 20243 raft_consensus.cc:493] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:41.359038 20243 raft_consensus.cc:3060] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:41.363889 20243 raft_consensus.cc:515] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.365234 20243 leader_election.cc:290] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 2 election: Requested vote from peers 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691), 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505)
I20251215 08:57:41.366206 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:41.366107 20317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954f43ae181e4c239a500a79d517d2b1" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:41.366736 20393 raft_consensus.cc:3060] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:41.366881 20317 raft_consensus.cc:3055] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:41.367269 20317 raft_consensus.cc:740] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:41.367944 20317 consensus_queue.cc:260] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.368976 20317 raft_consensus.cc:3060] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:41.371059 20393 raft_consensus.cc:2468] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 2.
I20251215 08:57:41.372076 20210 leader_election.cc:304] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:41.372903 20596 raft_consensus.cc:2804] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:41.373394 20596 raft_consensus.cc:697] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:41.374431 20317 raft_consensus.cc:2468] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 2.
I20251215 08:57:41.374289 20596 consensus_queue.cc:237] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.381382 20138 catalog_manager.cc:5654] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 1 to 2, leader changed from 65b535a7151242c7b67512ae19f24375 (127.19.163.66) to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 2 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:41.461833 20590 raft_consensus.cc:993] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375: : Instructing follower 7094b3ef76d943deb6fb0e997921ee2d to start an election
I20251215 08:57:41.462275 20479 raft_consensus.cc:1081] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Signalling peer 7094b3ef76d943deb6fb0e997921ee2d to start an election
I20251215 08:57:41.463559 20243 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585"
dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
from {username='slave'} at 127.0.0.1:35162
I20251215 08:57:41.463972 20243 raft_consensus.cc:493] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:41.464234 20243 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.468472 20243 raft_consensus.cc:515] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:41.469858 20243 leader_election.cc:290] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 3 election: Requested vote from peers 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67:36505), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:41.470685 20393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
I20251215 08:57:41.470810 20317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd0fca4ba614b3091bb89126c86e585" candidate_uuid: "7094b3ef76d943deb6fb0e997921ee2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:41.471140 20393 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.471302 20317 raft_consensus.cc:3055] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:57:41.471622 20317 raft_consensus.cc:740] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:41.472270 20317 consensus_queue.cc:260] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:41.473238 20317 raft_consensus.cc:3060] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.475167 20393 raft_consensus.cc:2468] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 3.
I20251215 08:57:41.476052 20210 leader_election.cc:304] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:41.476940 20596 raft_consensus.cc:2804] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:57:41.477407 20596 raft_consensus.cc:697] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 LEADER]: Becoming Leader. State: Replica: 7094b3ef76d943deb6fb0e997921ee2d, State: Running, Role: LEADER
I20251215 08:57:41.478457 20317 raft_consensus.cc:2468] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7094b3ef76d943deb6fb0e997921ee2d in term 3.
I20251215 08:57:41.478247 20596 consensus_queue.cc:237] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } }
I20251215 08:57:41.484730 20138 catalog_manager.cc:5654] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d reported cstate change: term changed from 2 to 3, leader changed from 65b535a7151242c7b67512ae19f24375 (127.19.163.66) to 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65). New cstate: current_term: 3 leader_uuid: "7094b3ef76d943deb6fb0e997921ee2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:41.664439 20590 raft_consensus.cc:993] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375: : Instructing follower 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:41.664897 20479 raft_consensus.cc:1081] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Signalling peer 829fd63ab8fc4c0ab6380914a02dc2d6 to start an election
I20251215 08:57:41.666219 20393 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "371033e1f37b435cb195960318735bdb"
dest_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6"
from {username='slave'} at 127.0.0.1:44308
I20251215 08:57:41.666688 20393 raft_consensus.cc:493] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:41.666924 20393 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.670953 20393 raft_consensus.cc:515] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.672219 20393 leader_election.cc:290] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 3 election: Requested vote from peers 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543), 65b535a7151242c7b67512ae19f24375 (127.19.163.66:45691)
I20251215 08:57:41.673033 20243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7094b3ef76d943deb6fb0e997921ee2d"
I20251215 08:57:41.673303 20317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371033e1f37b435cb195960318735bdb" candidate_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "65b535a7151242c7b67512ae19f24375"
I20251215 08:57:41.673645 20243 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.673971 20317 raft_consensus.cc:3055] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:57:41.674224 20317 raft_consensus.cc:740] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65b535a7151242c7b67512ae19f24375, State: Running, Role: LEADER
I20251215 08:57:41.674836 20317 consensus_queue.cc:260] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.675757 20317 raft_consensus.cc:3060] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:41.677891 20243 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 3.
I20251215 08:57:41.678828 20356 leader_election.cc:304] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7094b3ef76d943deb6fb0e997921ee2d, 829fd63ab8fc4c0ab6380914a02dc2d6; no voters:
I20251215 08:57:41.679585 20560 raft_consensus.cc:2804] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:57:41.680014 20560 raft_consensus.cc:697] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 LEADER]: Becoming Leader. State: Replica: 829fd63ab8fc4c0ab6380914a02dc2d6, State: Running, Role: LEADER
I20251215 08:57:41.680866 20317 raft_consensus.cc:2468] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829fd63ab8fc4c0ab6380914a02dc2d6 in term 3.
I20251215 08:57:41.680737 20560 consensus_queue.cc:237] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } }
I20251215 08:57:41.687203 20138 catalog_manager.cc:5654] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 reported cstate change: term changed from 2 to 3, leader changed from 65b535a7151242c7b67512ae19f24375 (127.19.163.66) to 829fd63ab8fc4c0ab6380914a02dc2d6 (127.19.163.67). New cstate: current_term: 3 leader_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:41.803758 20243 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 5 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251215 08:57:41.804814 20560 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:57:41.813189 20316 raft_consensus.cc:1275] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Refusing update from remote peer 7094b3ef76d943deb6fb0e997921ee2d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:41.814500 20317 raft_consensus.cc:1275] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 5 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251215 08:57:41.814569 20596 consensus_queue.cc:1048] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:41.816208 20592 consensus_queue.cc:1048] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:57:41.827607 20316 raft_consensus.cc:1275] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Refusing update from remote peer 7094b3ef76d943deb6fb0e997921ee2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:41.829329 20596 consensus_queue.cc:1048] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:41.832796 20393 raft_consensus.cc:1275] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Refusing update from remote peer 7094b3ef76d943deb6fb0e997921ee2d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:41.834558 20612 consensus_queue.cc:1048] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:57:41.853977 20393 raft_consensus.cc:1275] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Refusing update from remote peer 7094b3ef76d943deb6fb0e997921ee2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:41.855568 20596 consensus_queue.cc:1048] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:41.920548 20317 raft_consensus.cc:1275] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Refusing update from remote peer 7094b3ef76d943deb6fb0e997921ee2d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:41.921759 20596 consensus_queue.cc:1048] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:41.930361 20390 raft_consensus.cc:1275] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Refusing update from remote peer 7094b3ef76d943deb6fb0e997921ee2d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:41.931841 20612 consensus_queue.cc:1048] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "829fd63ab8fc4c0ab6380914a02dc2d6" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:42.197152 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:42.202013 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:42.202369 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:42.223866 20317 raft_consensus.cc:1275] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:42.225046 20592 consensus_queue.cc:1048] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b535a7151242c7b67512ae19f24375" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:42.232991 20244 raft_consensus.cc:1275] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Refusing update from remote peer 829fd63ab8fc4c0ab6380914a02dc2d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:42.234249 20592 consensus_queue.cc:1048] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7094b3ef76d943deb6fb0e997921ee2d" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:43.203462 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:43.208205 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:43.208608 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:44.209672 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:44.214751 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:44.215166 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:45.216246 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:45.221050 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:45.221465 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:46.222481 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:46.227411 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:46.227855 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:47.228865 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:47.233701 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:47.234104 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:48.235132 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:48.240562 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:48.241035 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:49.242192 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:49.247117 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:49.247535 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:50.248549 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:50.254012 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:50.254405 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:51.305490 20109 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20251215 08:57:51.306529 20109 tablet_server.cc:178] TabletServer@127.19.163.65:0 shutting down...
I20251215 08:57:51.335135 20109 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 08:57:51.336092 20109 tablet_replica.cc:333] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.336992 20109 raft_consensus.cc:2243] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.337670 20109 raft_consensus.cc:2272] T 40fa4688edda40508124f1077bdc390f P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.340090 20109 tablet_replica.cc:333] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.340669 20109 raft_consensus.cc:2243] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.341176 20109 raft_consensus.cc:2272] T 3000909d86d64aa8a3b265ce0fb0a913 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.342921 20109 tablet_replica.cc:333] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.343391 20109 raft_consensus.cc:2243] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.344301 20109 raft_consensus.cc:2272] T 954f43ae181e4c239a500a79d517d2b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.346414 20109 tablet_replica.cc:333] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.346997 20109 raft_consensus.cc:2243] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 5 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.347452 20109 raft_consensus.cc:2272] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 7094b3ef76d943deb6fb0e997921ee2d [term 5 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.349071 20109 tablet_replica.cc:333] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.349534 20109 raft_consensus.cc:2243] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.350291 20109 raft_consensus.cc:2272] T fdd0fca4ba614b3091bb89126c86e585 P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.351971 20109 tablet_replica.cc:333] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.352384 20109 raft_consensus.cc:2243] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.352788 20109 raft_consensus.cc:2272] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.354590 20109 tablet_replica.cc:333] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.355046 20109 raft_consensus.cc:2243] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.355430 20109 raft_consensus.cc:2272] T cc0a711ba0be47e5ae07f94c30d64b72 P 7094b3ef76d943deb6fb0e997921ee2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.356987 20109 tablet_replica.cc:333] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.357474 20109 raft_consensus.cc:2243] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.358351 20109 raft_consensus.cc:2272] T 14b54d338b7a4703b79365781fbd008b P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.360162 20109 tablet_replica.cc:333] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d: stopping tablet replica
I20251215 08:57:51.360592 20109 raft_consensus.cc:2243] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.361125 20109 raft_consensus.cc:2272] T 371033e1f37b435cb195960318735bdb P 7094b3ef76d943deb6fb0e997921ee2d [term 3 FOLLOWER]: Raft consensus is shut down!
W20251215 08:57:51.488257 20356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.163.65:44543: connect: Connection refused (error 111)
W20251215 08:57:51.494105 20356 consensus_peers.cc:597] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 -> Peer 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543): Couldn't send request to peer 7094b3ef76d943deb6fb0e997921ee2d. Status: Network error: Client connection negotiation failed: client connection to 127.19.163.65:44543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251215 08:57:51.494805 20282 consensus_peers.cc:597] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 -> Peer 7094b3ef76d943deb6fb0e997921ee2d (127.19.163.65:44543): Couldn't send request to peer 7094b3ef76d943deb6fb0e997921ee2d. Status: Network error: Client connection negotiation failed: client connection to 127.19.163.65:44543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251215 08:57:51.497967 20109 tablet_server.cc:195] TabletServer@127.19.163.65:0 shutdown complete.
I20251215 08:57:51.516626 20109 tablet_server.cc:178] TabletServer@127.19.163.66:0 shutting down...
I20251215 08:57:51.543972 20109 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 08:57:51.544664 20109 tablet_replica.cc:333] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.545332 20109 raft_consensus.cc:2243] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.545881 20109 raft_consensus.cc:2272] T 3000909d86d64aa8a3b265ce0fb0a913 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.547719 20109 tablet_replica.cc:333] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.548223 20109 raft_consensus.cc:2243] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.548692 20109 raft_consensus.cc:2272] T 954f43ae181e4c239a500a79d517d2b1 P 65b535a7151242c7b67512ae19f24375 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.550403 20109 tablet_replica.cc:333] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.550868 20109 raft_consensus.cc:2243] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.551592 20109 raft_consensus.cc:2272] T 40fa4688edda40508124f1077bdc390f P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.553301 20109 tablet_replica.cc:333] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.553797 20109 raft_consensus.cc:2243] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.554188 20109 raft_consensus.cc:2272] T 371033e1f37b435cb195960318735bdb P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.555742 20109 tablet_replica.cc:333] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.556197 20109 raft_consensus.cc:2243] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.556644 20109 raft_consensus.cc:2272] T 14b54d338b7a4703b79365781fbd008b P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.558311 20109 tablet_replica.cc:333] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.558753 20109 raft_consensus.cc:2243] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 1 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.559532 20109 raft_consensus.cc:2272] T cc0a711ba0be47e5ae07f94c30d64b72 P 65b535a7151242c7b67512ae19f24375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.561180 20109 tablet_replica.cc:333] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.561636 20109 raft_consensus.cc:2243] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.562404 20109 raft_consensus.cc:2272] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.563925 20109 tablet_replica.cc:333] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.564343 20109 raft_consensus.cc:2243] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.564723 20109 raft_consensus.cc:2272] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 65b535a7151242c7b67512ae19f24375 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.566232 20109 tablet_replica.cc:333] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375: stopping tablet replica
I20251215 08:57:51.566676 20109 raft_consensus.cc:2243] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.567135 20109 raft_consensus.cc:2272] T fdd0fca4ba614b3091bb89126c86e585 P 65b535a7151242c7b67512ae19f24375 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.593884 20109 tablet_server.cc:195] TabletServer@127.19.163.66:0 shutdown complete.
I20251215 08:57:51.611558 20109 tablet_server.cc:178] TabletServer@127.19.163.67:0 shutting down...
I20251215 08:57:51.637686 20109 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 08:57:51.638430 20109 tablet_replica.cc:333] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.639122 20109 raft_consensus.cc:2243] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.639647 20109 raft_consensus.cc:2272] T 954f43ae181e4c239a500a79d517d2b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.641582 20109 tablet_replica.cc:333] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.642102 20109 raft_consensus.cc:2243] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.642926 20109 raft_consensus.cc:2272] T 3000909d86d64aa8a3b265ce0fb0a913 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.644526 20109 tablet_replica.cc:333] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.644992 20109 raft_consensus.cc:2243] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.645401 20109 raft_consensus.cc:2272] T 40fa4688edda40508124f1077bdc390f P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.647024 20109 tablet_replica.cc:333] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.647482 20109 raft_consensus.cc:2243] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.648216 20109 raft_consensus.cc:2272] T 371033e1f37b435cb195960318735bdb P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.649834 20109 tablet_replica.cc:333] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.650291 20109 raft_consensus.cc:2243] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.650696 20109 raft_consensus.cc:2272] T cc0a711ba0be47e5ae07f94c30d64b72 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.652220 20109 tablet_replica.cc:333] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.652673 20109 raft_consensus.cc:2243] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.653193 20109 raft_consensus.cc:2272] T 14b54d338b7a4703b79365781fbd008b P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.654649 20109 tablet_replica.cc:333] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.655074 20109 raft_consensus.cc:2243] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 5 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.655813 20109 raft_consensus.cc:2272] T 6d2a63a43b2c4a4a80f94a4083c343b1 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.657392 20109 tablet_replica.cc:333] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.657819 20109 raft_consensus.cc:2243] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.658244 20109 raft_consensus.cc:2272] T fdd0fca4ba614b3091bb89126c86e585 P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.659869 20109 tablet_replica.cc:333] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6: stopping tablet replica
I20251215 08:57:51.660260 20109 raft_consensus.cc:2243] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:57:51.660671 20109 raft_consensus.cc:2272] T d6eb6ebde2ed48a2a28e6ca4fb98e9ed P 829fd63ab8fc4c0ab6380914a02dc2d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.686800 20109 tablet_server.cc:195] TabletServer@127.19.163.67:0 shutdown complete.
I20251215 08:57:51.705237 20109 master.cc:561] Master@127.19.163.126:44433 shutting down...
I20251215 08:57:51.846261 20109 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 1 LEADER]: Raft consensus shutting down.
I20251215 08:57:51.846957 20109 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:57:51.847266 20109 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8262533d139f4b09abad3556fe178493: stopping tablet replica
I20251215 08:57:51.866897 20109 master.cc:583] Master@127.19.163.126:44433 shutdown complete.
65b535a7151242c7b67512ae19f24375 4
7094b3ef76d943deb6fb0e997921ee2d 4
829fd63ab8fc4c0ab6380914a02dc2d6 1
65b535a7151242c7b67512ae19f24375 3
7094b3ef76d943deb6fb0e997921ee2d 3
829fd63ab8fc4c0ab6380914a02dc2d6 3
65b535a7151242c7b67512ae19f24375 8
7094b3ef76d943deb6fb0e997921ee2d 0
829fd63ab8fc4c0ab6380914a02dc2d6 1
65b535a7151242c7b67512ae19f24375 3
7094b3ef76d943deb6fb0e997921ee2d 3
829fd63ab8fc4c0ab6380914a02dc2d6 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30741 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20251215 08:57:51.907282 20109 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.163.126:38021
I20251215 08:57:51.908459 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:51.914628 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:51.914767 20635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:51.914688 20633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:51.916981 20109 server_base.cc:1047] running on GCE node
I20251215 08:57:51.917868 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:51.918076 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:51.918218 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789071918200 us; error 0 us; skew 500 ppm
I20251215 08:57:51.918735 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:51.920986 20109 webserver.cc:492] Webserver started at http://127.19.163.126:39465/ using document root <none> and password file <none>
I20251215 08:57:51.921535 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:51.921730 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:51.921967 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:51.923113 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/master-0-root/instance:
uuid: "742723155f494a569008014b9e0c3e75"
format_stamp: "Formatted at 2025-12-15 08:57:51 on dist-test-slave-s29j"
I20251215 08:57:51.927563 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20251215 08:57:51.930919 20640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:51.931700 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 08:57:51.931929 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/master-0-root
uuid: "742723155f494a569008014b9e0c3e75"
format_stamp: "Formatted at 2025-12-15 08:57:51 on dist-test-slave-s29j"
I20251215 08:57:51.932168 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:51.942999 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:51.944144 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:51.982501 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.126:38021
I20251215 08:57:51.982630 20691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.126:38021 every 8 connection(s)
I20251215 08:57:51.987054 20692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:51.996721 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75: Bootstrap starting.
I20251215 08:57:52.001390 20692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.005563 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75: No bootstrap required, opened a new log
I20251215 08:57:52.007622 20692 raft_consensus.cc:359] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742723155f494a569008014b9e0c3e75" member_type: VOTER }
I20251215 08:57:52.008071 20692 raft_consensus.cc:385] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.008287 20692 raft_consensus.cc:740] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742723155f494a569008014b9e0c3e75, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.008798 20692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742723155f494a569008014b9e0c3e75" member_type: VOTER }
I20251215 08:57:52.009416 20692 raft_consensus.cc:399] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 08:57:52.009661 20692 raft_consensus.cc:493] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 08:57:52.009945 20692 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:52.014010 20692 raft_consensus.cc:515] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742723155f494a569008014b9e0c3e75" member_type: VOTER }
I20251215 08:57:52.014519 20692 leader_election.cc:304] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742723155f494a569008014b9e0c3e75; no voters:
I20251215 08:57:52.015630 20692 leader_election.cc:290] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 08:57:52.015919 20695 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:52.017244 20695 raft_consensus.cc:697] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 1 LEADER]: Becoming Leader. State: Replica: 742723155f494a569008014b9e0c3e75, State: Running, Role: LEADER
I20251215 08:57:52.017966 20695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742723155f494a569008014b9e0c3e75" member_type: VOTER }
I20251215 08:57:52.018642 20692 sys_catalog.cc:565] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 08:57:52.023573 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742723155f494a569008014b9e0c3e75. Latest consensus state: current_term: 1 leader_uuid: "742723155f494a569008014b9e0c3e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742723155f494a569008014b9e0c3e75" member_type: VOTER } }
I20251215 08:57:52.024319 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [sys.catalog]: This master's current role is: LEADER
I20251215 08:57:52.024885 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742723155f494a569008014b9e0c3e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742723155f494a569008014b9e0c3e75" member_type: VOTER } }
I20251215 08:57:52.025326 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [sys.catalog]: This master's current role is: LEADER
I20251215 08:57:52.026530 20703 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 08:57:52.031528 20703 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 08:57:52.035005 20109 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 08:57:52.040733 20703 catalog_manager.cc:1357] Generated new cluster ID: 562aabc7f5fe4d8aa59f05078ca9d1c6
I20251215 08:57:52.041036 20703 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 08:57:52.054903 20703 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 08:57:52.056107 20703 catalog_manager.cc:1514] Loading token signing keys...
I20251215 08:57:52.069141 20703 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75: Generated new TSK 0
I20251215 08:57:52.069759 20703 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 08:57:52.102367 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:52.108717 20713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:52.108968 20714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:52.110440 20716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:52.111649 20109 server_base.cc:1047] running on GCE node
I20251215 08:57:52.112428 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:52.112602 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:52.112735 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789072112723 us; error 0 us; skew 500 ppm
I20251215 08:57:52.113235 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:52.115513 20109 webserver.cc:492] Webserver started at http://127.19.163.65:41003/ using document root <none> and password file <none>
I20251215 08:57:52.116006 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:52.116168 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:52.116398 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:52.117515 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-0-root/instance:
uuid: "2b0986965d0d49bf87a9aa87d2f06643"
format_stamp: "Formatted at 2025-12-15 08:57:52 on dist-test-slave-s29j"
I20251215 08:57:52.122295 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251215 08:57:52.125545 20721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.126343 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 08:57:52.126693 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-0-root
uuid: "2b0986965d0d49bf87a9aa87d2f06643"
format_stamp: "Formatted at 2025-12-15 08:57:52 on dist-test-slave-s29j"
I20251215 08:57:52.126977 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:52.137212 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:52.138429 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:52.139860 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:57:52.142130 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:57:52.142320 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.142554 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:57:52.142706 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.182536 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.65:43261
I20251215 08:57:52.182641 20783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.65:43261 every 8 connection(s)
I20251215 08:57:52.187006 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:52.194512 20788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:52.198024 20791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:52.198158 20789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:52.199857 20109 server_base.cc:1047] running on GCE node
I20251215 08:57:52.200878 20784 heartbeater.cc:344] Connected to a master server at 127.19.163.126:38021
I20251215 08:57:52.201330 20784 heartbeater.cc:461] Registering TS with master...
I20251215 08:57:52.201614 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:52.201872 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:52.202062 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789072202044 us; error 0 us; skew 500 ppm
I20251215 08:57:52.202216 20784 heartbeater.cc:507] Master 127.19.163.126:38021 requested a full tablet report, sending...
I20251215 08:57:52.202791 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:52.204573 20657 ts_manager.cc:194] Registered new tserver with Master: 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261)
I20251215 08:57:52.205530 20109 webserver.cc:492] Webserver started at http://127.19.163.66:36119/ using document root <none> and password file <none>
I20251215 08:57:52.206017 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:52.206202 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:52.206458 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:52.206501 20657 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56448
I20251215 08:57:52.207783 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-1-root/instance:
uuid: "6282ff6904b948e59e40f2db52b2be83"
format_stamp: "Formatted at 2025-12-15 08:57:52 on dist-test-slave-s29j"
I20251215 08:57:52.212550 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20251215 08:57:52.215673 20796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.216373 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 08:57:52.216655 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-1-root
uuid: "6282ff6904b948e59e40f2db52b2be83"
format_stamp: "Formatted at 2025-12-15 08:57:52 on dist-test-slave-s29j"
I20251215 08:57:52.216930 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:52.241101 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:52.242316 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:52.243808 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:57:52.246027 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:57:52.246229 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.246507 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:57:52.246665 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.286288 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.66:45293
I20251215 08:57:52.286370 20858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.66:45293 every 8 connection(s)
I20251215 08:57:52.290844 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:57:52.297410 20862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:52.298090 20863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:57:52.301033 20865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:57:52.301558 20109 server_base.cc:1047] running on GCE node
I20251215 08:57:52.302663 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:57:52.302867 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:57:52.303074 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789072303054 us; error 0 us; skew 500 ppm
I20251215 08:57:52.303498 20859 heartbeater.cc:344] Connected to a master server at 127.19.163.126:38021
I20251215 08:57:52.303737 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:57:52.303906 20859 heartbeater.cc:461] Registering TS with master...
I20251215 08:57:52.304828 20859 heartbeater.cc:507] Master 127.19.163.126:38021 requested a full tablet report, sending...
I20251215 08:57:52.306689 20109 webserver.cc:492] Webserver started at http://127.19.163.67:42581/ using document root <none> and password file <none>
I20251215 08:57:52.307107 20657 ts_manager.cc:194] Registered new tserver with Master: 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:52.307371 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:57:52.307632 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:57:52.307935 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:57:52.308909 20657 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56458
I20251215 08:57:52.309540 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-2-root/instance:
uuid: "48d193db12ff4e5b973328011a578553"
format_stamp: "Formatted at 2025-12-15 08:57:52 on dist-test-slave-s29j"
I20251215 08:57:52.314445 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251215 08:57:52.317654 20870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.318413 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 08:57:52.318711 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-2-root
uuid: "48d193db12ff4e5b973328011a578553"
format_stamp: "Formatted at 2025-12-15 08:57:52 on dist-test-slave-s29j"
I20251215 08:57:52.318980 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765789040996898-20109-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:57:52.350176 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:57:52.351361 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:57:52.352814 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:57:52.354992 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:57:52.355192 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.355434 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:57:52.355589 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:57:52.395076 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.67:41687
I20251215 08:57:52.395179 20932 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.67:41687 every 8 connection(s)
I20251215 08:57:52.412127 20933 heartbeater.cc:344] Connected to a master server at 127.19.163.126:38021
I20251215 08:57:52.412487 20933 heartbeater.cc:461] Registering TS with master...
I20251215 08:57:52.413272 20933 heartbeater.cc:507] Master 127.19.163.126:38021 requested a full tablet report, sending...
I20251215 08:57:52.415143 20657 ts_manager.cc:194] Registered new tserver with Master: 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:52.415321 20109 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017183883s
I20251215 08:57:52.416975 20657 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56460
I20251215 08:57:52.418650 20109 test_util.cc:276] Using random seed: -831985869
I20251215 08:57:52.442155 20657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56462:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 08:57:52.444932 20657 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 08:57:52.514978 20748 tablet_service.cc:1505] Processing CreateTablet for tablet dc67dabaa62248d28af779466ba95976 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251215 08:57:52.515549 20747 tablet_service.cc:1505] Processing CreateTablet for tablet 1b8468317e1e402aba91c174231624e2 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251215 08:57:52.516531 20748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc67dabaa62248d28af779466ba95976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.517973 20746 tablet_service.cc:1505] Processing CreateTablet for tablet ccaea97f463c4731b8e73ee3d0028807 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251215 08:57:52.518196 20749 tablet_service.cc:1505] Processing CreateTablet for tablet afd4e1cf18544f84a1a61cb7df061b74 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251215 08:57:52.519428 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccaea97f463c4731b8e73ee3d0028807. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.521196 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd4e1cf18544f84a1a61cb7df061b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.527109 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8468317e1e402aba91c174231624e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.527609 20745 tablet_service.cc:1505] Processing CreateTablet for tablet a34845828f9a42118a523af87e308b08 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251215 08:57:52.528975 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34845828f9a42118a523af87e308b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.530939 20744 tablet_service.cc:1505] Processing CreateTablet for tablet a058316801f041a898362d1fdd5db080 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251215 08:57:52.532380 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a058316801f041a898362d1fdd5db080. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.539872 20743 tablet_service.cc:1505] Processing CreateTablet for tablet fd7013ed60464deca91ae876affc032d (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251215 08:57:52.541381 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7013ed60464deca91ae876affc032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.559620 20947 tablet_bootstrap.cc:492] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:52.561937 20822 tablet_service.cc:1505] Processing CreateTablet for tablet 1b8468317e1e402aba91c174231624e2 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251215 08:57:52.563373 20822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8468317e1e402aba91c174231624e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.563303 20824 tablet_service.cc:1505] Processing CreateTablet for tablet afd4e1cf18544f84a1a61cb7df061b74 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251215 08:57:52.565686 20821 tablet_service.cc:1505] Processing CreateTablet for tablet ccaea97f463c4731b8e73ee3d0028807 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251215 08:57:52.566380 20823 tablet_service.cc:1505] Processing CreateTablet for tablet dc67dabaa62248d28af779466ba95976 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251215 08:57:52.567970 20820 tablet_service.cc:1505] Processing CreateTablet for tablet a34845828f9a42118a523af87e308b08 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251215 08:57:52.570223 20819 tablet_service.cc:1505] Processing CreateTablet for tablet a058316801f041a898362d1fdd5db080 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251215 08:57:52.572503 20818 tablet_service.cc:1505] Processing CreateTablet for tablet fd7013ed60464deca91ae876affc032d (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251215 08:57:52.574748 20745 tablet_service.cc:1505] Processing CreateTablet for tablet 691a85c9bd184f6788b51c19d2ad7273 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251215 08:57:52.576102 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691a85c9bd184f6788b51c19d2ad7273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.576494 20897 tablet_service.cc:1505] Processing CreateTablet for tablet dc67dabaa62248d28af779466ba95976 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251215 08:57:52.577759 20897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc67dabaa62248d28af779466ba95976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.578197 20898 tablet_service.cc:1505] Processing CreateTablet for tablet afd4e1cf18544f84a1a61cb7df061b74 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251215 08:57:52.579452 20898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd4e1cf18544f84a1a61cb7df061b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.583281 20824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd4e1cf18544f84a1a61cb7df061b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.583617 20895 tablet_service.cc:1505] Processing CreateTablet for tablet ccaea97f463c4731b8e73ee3d0028807 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251215 08:57:52.584982 20895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccaea97f463c4731b8e73ee3d0028807. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.585408 20821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccaea97f463c4731b8e73ee3d0028807. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.587162 20819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a058316801f041a898362d1fdd5db080. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.589049 20947 tablet_bootstrap.cc:654] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.590265 20894 tablet_service.cc:1505] Processing CreateTablet for tablet a34845828f9a42118a523af87e308b08 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251215 08:57:52.590648 20893 tablet_service.cc:1505] Processing CreateTablet for tablet a058316801f041a898362d1fdd5db080 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251215 08:57:52.591627 20894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34845828f9a42118a523af87e308b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.582554 20896 tablet_service.cc:1505] Processing CreateTablet for tablet 1b8468317e1e402aba91c174231624e2 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251215 08:57:52.593063 20896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8468317e1e402aba91c174231624e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.593557 20820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34845828f9a42118a523af87e308b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.594275 20823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc67dabaa62248d28af779466ba95976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.597401 20893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a058316801f041a898362d1fdd5db080. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.602825 20892 tablet_service.cc:1505] Processing CreateTablet for tablet fd7013ed60464deca91ae876affc032d (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251215 08:57:52.604136 20892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7013ed60464deca91ae876affc032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.601534 20818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7013ed60464deca91ae876affc032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.636471 20817 tablet_service.cc:1505] Processing CreateTablet for tablet 691a85c9bd184f6788b51c19d2ad7273 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251215 08:57:52.637908 20817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691a85c9bd184f6788b51c19d2ad7273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.646838 20745 tablet_service.cc:1505] Processing CreateTablet for tablet 6807ea1cd51f4491aaa320e8382f23e1 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251215 08:57:52.648600 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6807ea1cd51f4491aaa320e8382f23e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.659013 20898 tablet_service.cc:1505] Processing CreateTablet for tablet 691a85c9bd184f6788b51c19d2ad7273 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251215 08:57:52.661839 20748 tablet_service.cc:1505] Processing CreateTablet for tablet 576271add9c246529e6deb72528494f1 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251215 08:57:52.663123 20748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576271add9c246529e6deb72528494f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.663532 20823 tablet_service.cc:1505] Processing CreateTablet for tablet 6807ea1cd51f4491aaa320e8382f23e1 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251215 08:57:52.664857 20823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6807ea1cd51f4491aaa320e8382f23e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.661294 20898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691a85c9bd184f6788b51c19d2ad7273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.667277 20891 tablet_service.cc:1505] Processing CreateTablet for tablet 6807ea1cd51f4491aaa320e8382f23e1 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251215 08:57:52.669090 20891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6807ea1cd51f4491aaa320e8382f23e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.691000 20816 tablet_service.cc:1505] Processing CreateTablet for tablet 576271add9c246529e6deb72528494f1 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251215 08:57:52.692875 20816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576271add9c246529e6deb72528494f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.689569 20949 tablet_bootstrap.cc:492] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:52.708705 20947 tablet_bootstrap.cc:492] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:52.709240 20947 ts_tablet_manager.cc:1403] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.150s user 0.019s sys 0.030s
I20251215 08:57:52.712007 20947 raft_consensus.cc:359] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.713886 20947 raft_consensus.cc:385] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.714262 20947 raft_consensus.cc:740] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.716898 20950 tablet_bootstrap.cc:492] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:52.721411 20947 consensus_queue.cc:260] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.722147 20892 tablet_service.cc:1505] Processing CreateTablet for tablet 576271add9c246529e6deb72528494f1 (DEFAULT_TABLE table=test-workload [id=6a6e8640c6b945fbb84370c5f7e336d9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251215 08:57:52.723843 20892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576271add9c246529e6deb72528494f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:57:52.734830 20950 tablet_bootstrap.cc:654] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.741408 20949 tablet_bootstrap.cc:654] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.746543 20784 heartbeater.cc:499] Master 127.19.163.126:38021 was elected leader, sending a full tablet report...
I20251215 08:57:52.747566 20947 ts_tablet_manager.cc:1434] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.038s user 0.011s sys 0.017s
I20251215 08:57:52.757369 20947 tablet_bootstrap.cc:492] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:52.766048 20947 tablet_bootstrap.cc:654] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.769294 20949 tablet_bootstrap.cc:492] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:52.769984 20949 ts_tablet_manager.cc:1403] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.081s user 0.016s sys 0.012s
I20251215 08:57:52.773633 20949 raft_consensus.cc:359] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.774716 20949 raft_consensus.cc:385] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.775174 20949 raft_consensus.cc:740] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.776373 20949 consensus_queue.cc:260] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.782158 20950 tablet_bootstrap.cc:492] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:52.782734 20950 ts_tablet_manager.cc:1403] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.066s user 0.022s sys 0.017s
I20251215 08:57:52.785707 20950 raft_consensus.cc:359] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.786583 20950 raft_consensus.cc:385] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.786957 20950 raft_consensus.cc:740] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.788276 20947 tablet_bootstrap.cc:492] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:52.787878 20950 consensus_queue.cc:260] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.788779 20947 ts_tablet_manager.cc:1403] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.006s
I20251215 08:57:52.792387 20859 heartbeater.cc:499] Master 127.19.163.126:38021 was elected leader, sending a full tablet report...
I20251215 08:57:52.792248 20947 raft_consensus.cc:359] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.792800 20950 ts_tablet_manager.cc:1434] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20251215 08:57:52.792999 20947 raft_consensus.cc:385] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.793318 20947 raft_consensus.cc:740] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.794029 20950 tablet_bootstrap.cc:492] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:52.794005 20947 consensus_queue.cc:260] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.796482 20947 ts_tablet_manager.cc:1434] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251215 08:57:52.798048 20933 heartbeater.cc:499] Master 127.19.163.126:38021 was elected leader, sending a full tablet report...
I20251215 08:57:52.798715 20949 ts_tablet_manager.cc:1434] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.028s user 0.019s sys 0.007s
I20251215 08:57:52.799700 20949 tablet_bootstrap.cc:492] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:52.801608 20951 raft_consensus.cc:493] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:52.802474 20950 tablet_bootstrap.cc:654] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.802107 20951 raft_consensus.cc:515] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.803052 20947 tablet_bootstrap.cc:492] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:52.806079 20949 tablet_bootstrap.cc:654] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.811115 20955 raft_consensus.cc:493] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:52.811610 20947 tablet_bootstrap.cc:654] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.811818 20955 raft_consensus.cc:515] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.814885 20951 leader_election.cc:290] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:52.829162 20955 leader_election.cc:290] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:52.831799 20947 tablet_bootstrap.cc:492] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:52.849651 20947 ts_tablet_manager.cc:1403] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.014s
I20251215 08:57:52.845480 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:52.847167 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
W20251215 08:57:52.855273 20872 leader_election.cc:343] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:57:52.855990 20874 leader_election.cc:343] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:52.856463 20874 leader_election.cc:304] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48d193db12ff4e5b973328011a578553; no voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:52.857626 20955 raft_consensus.cc:2749] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:52.855881 20947 raft_consensus.cc:359] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.858132 20947 raft_consensus.cc:385] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.858556 20947 raft_consensus.cc:740] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.859483 20947 consensus_queue.cc:260] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.862705 20947 ts_tablet_manager.cc:1434] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20251215 08:57:52.863968 20947 tablet_bootstrap.cc:492] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:52.875737 20947 tablet_bootstrap.cc:654] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.880774 20949 tablet_bootstrap.cc:492] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:52.881492 20949 ts_tablet_manager.cc:1403] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.082s user 0.028s sys 0.053s
I20251215 08:57:52.887940 20950 tablet_bootstrap.cc:492] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:52.888669 20950 ts_tablet_manager.cc:1403] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.095s user 0.033s sys 0.028s
I20251215 08:57:52.890316 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
W20251215 08:57:52.891883 20725 leader_election.cc:343] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:52.892577 20950 raft_consensus.cc:359] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.893602 20950 raft_consensus.cc:385] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.893994 20950 raft_consensus.cc:740] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.894913 20949 raft_consensus.cc:359] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.895874 20949 raft_consensus.cc:385] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.896209 20949 raft_consensus.cc:740] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.896924 20949 consensus_queue.cc:260] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.898023 20950 consensus_queue.cc:260] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.900391 20950 ts_tablet_manager.cc:1434] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.011s user 0.001s sys 0.004s
I20251215 08:57:52.901369 20950 tablet_bootstrap.cc:492] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:52.902235 20947 tablet_bootstrap.cc:492] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:52.902688 20947 ts_tablet_manager.cc:1403] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.015s
I20251215 08:57:52.905563 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:52.905970 20947 raft_consensus.cc:359] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.906684 20947 raft_consensus.cc:385] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.907038 20947 raft_consensus.cc:740] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.907531 20950 tablet_bootstrap.cc:654] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
W20251215 08:57:52.907744 20725 leader_election.cc:343] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:52.907949 20947 consensus_queue.cc:260] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.908401 20725 leader_election.cc:304] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643; no voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:52.909618 20951 raft_consensus.cc:2749] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:52.910723 20949 ts_tablet_manager.cc:1434] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20251215 08:57:52.911640 20949 tablet_bootstrap.cc:492] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:52.915648 20947 ts_tablet_manager.cc:1434] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20251215 08:57:52.916872 20947 tablet_bootstrap.cc:492] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:52.918145 20949 tablet_bootstrap.cc:654] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.926190 20951 raft_consensus.cc:493] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:52.926774 20947 tablet_bootstrap.cc:654] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.926733 20951 raft_consensus.cc:515] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.928890 20951 leader_election.cc:290] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:52.930495 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:52.930672 20949 tablet_bootstrap.cc:492] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:52.930850 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:52.931443 20949 ts_tablet_manager.cc:1403] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
W20251215 08:57:52.931843 20725 leader_election.cc:343] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:57:52.932545 20725 leader_election.cc:343] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:52.932941 20725 leader_election.cc:304] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643; no voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:52.933764 20951 raft_consensus.cc:2749] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:52.933527 20950 tablet_bootstrap.cc:492] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:52.935580 20950 ts_tablet_manager.cc:1403] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.000s
I20251215 08:57:52.935685 20949 raft_consensus.cc:359] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.936790 20949 raft_consensus.cc:385] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.937275 20949 raft_consensus.cc:740] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.938454 20949 consensus_queue.cc:260] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.941789 20949 ts_tablet_manager.cc:1434] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20251215 08:57:52.941689 20950 raft_consensus.cc:359] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.942556 20950 raft_consensus.cc:385] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.942912 20950 raft_consensus.cc:740] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.943028 20949 tablet_bootstrap.cc:492] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:52.943691 20950 consensus_queue.cc:260] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.944206 20947 tablet_bootstrap.cc:492] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:52.944801 20947 ts_tablet_manager.cc:1403] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.020s
I20251215 08:57:52.946097 20950 ts_tablet_manager.cc:1434] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251215 08:57:52.947042 20950 tablet_bootstrap.cc:492] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:52.947417 20956 raft_consensus.cc:493] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:52.947510 20947 raft_consensus.cc:359] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.948269 20947 raft_consensus.cc:385] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:52.947911 20956 raft_consensus.cc:515] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.948573 20947 raft_consensus.cc:740] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:52.949371 20947 consensus_queue.cc:260] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.953667 20950 tablet_bootstrap.cc:654] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.953845 20947 ts_tablet_manager.cc:1434] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20251215 08:57:52.954964 20947 tablet_bootstrap.cc:492] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:52.958434 20955 raft_consensus.cc:493] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:52.959300 20955 raft_consensus.cc:515] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.958236 20956 leader_election.cc:290] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:52.961298 20955 leader_election.cc:290] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:52.962455 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:52.962906 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:52.963307 20759 raft_consensus.cc:2468] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
W20251215 08:57:52.964226 20874 leader_election.cc:343] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:52.966346 20949 tablet_bootstrap.cc:654] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.971057 20955 raft_consensus.cc:493] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:52.971591 20955 raft_consensus.cc:515] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.973634 20955 leader_election.cc:290] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:52.973908 20872 leader_election.cc:304] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters: 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:52.975086 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:52.975703 20834 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
I20251215 08:57:52.976608 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:52.977037 20874 leader_election.cc:304] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:52.981597 20947 tablet_bootstrap.cc:654] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:52.978705 20955 raft_consensus.cc:2804] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:52.983561 20955 raft_consensus.cc:493] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:52.983892 20955 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:57:52.986305 20872 leader_election.cc:343] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:52.995182 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:52.995999 20759 raft_consensus.cc:2468] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 0.
I20251215 08:57:52.996253 20947 tablet_bootstrap.cc:492] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:52.996737 20947 ts_tablet_manager.cc:1403] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.000s
I20251215 08:57:52.997128 20971 raft_consensus.cc:2804] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:52.997632 20971 raft_consensus.cc:493] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:52.997467 20798 leader_election.cc:304] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:52.998529 20956 raft_consensus.cc:2804] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:52.998880 20956 raft_consensus.cc:493] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:52.999188 20956 raft_consensus.cc:3060] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:52.999282 20947 raft_consensus.cc:359] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:52.999872 20947 raft_consensus.cc:385] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.000097 20947 raft_consensus.cc:740] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.000152 20955 raft_consensus.cc:515] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.000720 20947 consensus_queue.cc:260] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.003151 20950 tablet_bootstrap.cc:492] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:53.006562 20950 ts_tablet_manager.cc:1403] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.060s user 0.024s sys 0.007s
I20251215 08:57:53.005314 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.007551 20971 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.007947 20759 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.008268 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
W20251215 08:57:53.009222 20874 leader_election.cc:343] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:53.006592 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:53.005489 20947 ts_tablet_manager.cc:1434] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
W20251215 08:57:53.014212 20800 leader_election.cc:343] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:53.014490 20947 tablet_bootstrap.cc:492] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:53.015921 20950 raft_consensus.cc:359] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.006790 20956 raft_consensus.cc:515] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.007365 20955 leader_election.cc:290] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.007076 20949 tablet_bootstrap.cc:492] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:53.016304 20759 raft_consensus.cc:2468] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.016883 20950 raft_consensus.cc:385] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.017372 20949 ts_tablet_manager.cc:1403] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.004s
I20251215 08:57:53.017382 20950 raft_consensus.cc:740] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.018056 20872 leader_election.cc:304] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters: 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:53.018451 20950 consensus_queue.cc:260] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.019191 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.019827 20758 raft_consensus.cc:3060] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.020982 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553"
W20251215 08:57:53.022341 20800 leader_election.cc:343] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:53.023937 20949 raft_consensus.cc:359] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.024638 20949 raft_consensus.cc:385] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.024942 20949 raft_consensus.cc:740] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.025164 20947 tablet_bootstrap.cc:654] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.025692 20955 raft_consensus.cc:2804] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.026722 20949 consensus_queue.cc:260] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.027179 20758 raft_consensus.cc:2468] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 1.
I20251215 08:57:53.028376 20798 leader_election.cc:304] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters: 48d193db12ff4e5b973328011a578553
W20251215 08:57:53.029960 20705 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:57:53.019207 20956 leader_election.cc:290] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.033489 20955 raft_consensus.cc:697] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:53.034359 20955 consensus_queue.cc:237] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.035900 20956 raft_consensus.cc:2804] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.036355 20956 raft_consensus.cc:697] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Becoming Leader. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:57:53.037173 20956 consensus_queue.cc:237] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.041307 20949 ts_tablet_manager.cc:1434] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.023s user 0.008s sys 0.001s
I20251215 08:57:53.042685 20949 tablet_bootstrap.cc:492] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:53.047516 20950 ts_tablet_manager.cc:1434] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.034s user 0.011s sys 0.012s
I20251215 08:57:53.051836 20950 tablet_bootstrap.cc:492] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:53.063208 20971 raft_consensus.cc:515] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.068285 20949 tablet_bootstrap.cc:654] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.069268 20950 tablet_bootstrap.cc:654] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.074086 20657 catalog_manager.cc:5654] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 1 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:53.075137 20654 catalog_manager.cc:5654] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "6282ff6904b948e59e40f2db52b2be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.078990 20971 leader_election.cc:290] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.079728 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
W20251215 08:57:53.081045 20872 leader_election.cc:343] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:57:53.080343 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.083178 20834 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.088102 20950 tablet_bootstrap.cc:492] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:53.088629 20950 ts_tablet_manager.cc:1403] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.010s
I20251215 08:57:53.091044 20834 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.091524 20950 raft_consensus.cc:359] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.092216 20950 raft_consensus.cc:385] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.092520 20950 raft_consensus.cc:740] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.092393 20874 leader_election.cc:304] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters: 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:57:53.093672 20971 raft_consensus.cc:2804] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.094336 20971 raft_consensus.cc:697] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:53.094026 20950 consensus_queue.cc:260] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.095784 20949 tablet_bootstrap.cc:492] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:53.096268 20949 ts_tablet_manager.cc:1403] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.007s
I20251215 08:57:53.095530 20971 consensus_queue.cc:237] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.096539 20950 ts_tablet_manager.cc:1434] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251215 08:57:53.097564 20950 tablet_bootstrap.cc:492] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:53.100404 20947 tablet_bootstrap.cc:492] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:53.100906 20947 ts_tablet_manager.cc:1403] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.087s user 0.024s sys 0.038s
I20251215 08:57:53.100917 20949 raft_consensus.cc:359] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.101891 20949 raft_consensus.cc:385] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.102206 20949 raft_consensus.cc:740] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.103044 20949 consensus_queue.cc:260] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.104029 20950 tablet_bootstrap.cc:654] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.104127 20947 raft_consensus.cc:359] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.104903 20947 raft_consensus.cc:385] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.105221 20947 raft_consensus.cc:740] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.105808 20949 ts_tablet_manager.cc:1434] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20251215 08:57:53.105958 20947 consensus_queue.cc:260] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.106877 20949 tablet_bootstrap.cc:492] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:53.108574 20947 ts_tablet_manager.cc:1434] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251215 08:57:53.114132 20947 tablet_bootstrap.cc:492] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:53.120815 20947 tablet_bootstrap.cc:654] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.130268 20654 catalog_manager.cc:5654] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 1 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:53.132545 20949 tablet_bootstrap.cc:654] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.132779 20947 tablet_bootstrap.cc:492] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:53.133263 20947 ts_tablet_manager.cc:1403] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20251215 08:57:53.135757 20947 raft_consensus.cc:359] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.136368 20947 raft_consensus.cc:385] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.136633 20947 raft_consensus.cc:740] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.137424 20950 tablet_bootstrap.cc:492] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:53.137413 20947 consensus_queue.cc:260] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.138841 20950 ts_tablet_manager.cc:1403] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.024s
I20251215 08:57:53.139884 20947 ts_tablet_manager.cc:1434] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251215 08:57:53.140762 20947 tablet_bootstrap.cc:492] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:53.141783 20950 raft_consensus.cc:359] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.142474 20950 raft_consensus.cc:385] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.142777 20950 raft_consensus.cc:740] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.143520 20950 consensus_queue.cc:260] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.145246 20949 tablet_bootstrap.cc:492] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:53.145716 20949 ts_tablet_manager.cc:1403] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.002s
I20251215 08:57:53.145869 20950 ts_tablet_manager.cc:1434] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20251215 08:57:53.146862 20950 tablet_bootstrap.cc:492] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:53.148183 20947 tablet_bootstrap.cc:654] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.148451 20949 raft_consensus.cc:359] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.149200 20949 raft_consensus.cc:385] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.149500 20949 raft_consensus.cc:740] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.152419 20949 consensus_queue.cc:260] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.153175 20950 tablet_bootstrap.cc:654] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.154882 20949 ts_tablet_manager.cc:1434] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251215 08:57:53.158325 20949 tablet_bootstrap.cc:492] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:53.163842 20947 tablet_bootstrap.cc:492] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:53.164566 20947 ts_tablet_manager.cc:1403] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.001s
I20251215 08:57:53.165721 20949 tablet_bootstrap.cc:654] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.167927 20947 raft_consensus.cc:359] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.168854 20947 raft_consensus.cc:385] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.169234 20947 raft_consensus.cc:740] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.170099 20947 consensus_queue.cc:260] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.172638 20947 ts_tablet_manager.cc:1434] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251215 08:57:53.173300 20950 tablet_bootstrap.cc:492] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:53.173555 20947 tablet_bootstrap.cc:492] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643: Bootstrap starting.
I20251215 08:57:53.173765 20950 ts_tablet_manager.cc:1403] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.015s
I20251215 08:57:53.174448 20949 tablet_bootstrap.cc:492] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:53.174971 20949 ts_tablet_manager.cc:1403] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20251215 08:57:53.176502 20950 raft_consensus.cc:359] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.177301 20950 raft_consensus.cc:385] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.177623 20950 raft_consensus.cc:740] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.178047 20949 raft_consensus.cc:359] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.178383 20950 consensus_queue.cc:260] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.178838 20949 raft_consensus.cc:385] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.179356 20947 tablet_bootstrap.cc:654] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.179258 20949 raft_consensus.cc:740] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.180289 20949 consensus_queue.cc:260] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.180871 20950 ts_tablet_manager.cc:1434] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251215 08:57:53.181983 20950 tablet_bootstrap.cc:492] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:53.183020 20949 ts_tablet_manager.cc:1434] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251215 08:57:53.184078 20949 tablet_bootstrap.cc:492] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:53.190241 20950 tablet_bootstrap.cc:654] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.191085 20949 tablet_bootstrap.cc:654] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.197059 20947 tablet_bootstrap.cc:492] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643: No bootstrap required, opened a new log
I20251215 08:57:53.197752 20947 ts_tablet_manager.cc:1403] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.005s
I20251215 08:57:53.201200 20947 raft_consensus.cc:359] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.202207 20947 raft_consensus.cc:385] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.202544 20947 raft_consensus.cc:740] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.203431 20947 consensus_queue.cc:260] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.205708 20947 ts_tablet_manager.cc:1434] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20251215 08:57:53.206743 20950 tablet_bootstrap.cc:492] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:53.207293 20950 ts_tablet_manager.cc:1403] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.004s
I20251215 08:57:53.210665 20950 raft_consensus.cc:359] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.211567 20950 raft_consensus.cc:385] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.211905 20950 raft_consensus.cc:740] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.212568 20950 consensus_queue.cc:260] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.214639 20950 ts_tablet_manager.cc:1434] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251215 08:57:53.217360 20950 tablet_bootstrap.cc:492] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:53.221375 20949 tablet_bootstrap.cc:492] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:53.221910 20949 ts_tablet_manager.cc:1403] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.003s
I20251215 08:57:53.223448 20950 tablet_bootstrap.cc:654] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.224550 20949 raft_consensus.cc:359] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.225360 20949 raft_consensus.cc:385] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.225693 20949 raft_consensus.cc:740] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.226454 20949 consensus_queue.cc:260] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.228430 20949 ts_tablet_manager.cc:1434] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:57:53.228852 20956 raft_consensus.cc:493] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.229363 20956 raft_consensus.cc:515] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.230561 20949 tablet_bootstrap.cc:492] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:53.231308 20950 tablet_bootstrap.cc:492] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:53.231290 20956 leader_election.cc:290] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.231881 20950 ts_tablet_manager.cc:1403] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20251215 08:57:53.232651 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:53.232870 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:53.233448 20758 raft_consensus.cc:2393] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6282ff6904b948e59e40f2db52b2be83 in current term 1: Already voted for candidate 48d193db12ff4e5b973328011a578553 in this term.
I20251215 08:57:53.234462 20798 leader_election.cc:304] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6282ff6904b948e59e40f2db52b2be83; no voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553
I20251215 08:57:53.234591 20950 raft_consensus.cc:359] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.235100 20956 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.235286 20950 raft_consensus.cc:385] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.235643 20950 raft_consensus.cc:740] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.236307 20950 consensus_queue.cc:260] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.238489 20950 ts_tablet_manager.cc:1434] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251215 08:57:53.239444 20950 tablet_bootstrap.cc:492] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83: Bootstrap starting.
I20251215 08:57:53.239773 20949 tablet_bootstrap.cc:654] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.246271 20950 tablet_bootstrap.cc:654] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.247175 20949 tablet_bootstrap.cc:492] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:53.247547 20949 ts_tablet_manager.cc:1403] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20251215 08:57:53.248327 20956 raft_consensus.cc:2749] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:53.249945 20949 raft_consensus.cc:359] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.250594 20949 raft_consensus.cc:385] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.250859 20949 raft_consensus.cc:740] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.251529 20949 consensus_queue.cc:260] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.252892 20950 tablet_bootstrap.cc:492] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83: No bootstrap required, opened a new log
I20251215 08:57:53.253350 20950 ts_tablet_manager.cc:1403] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20251215 08:57:53.253717 20949 ts_tablet_manager.cc:1434] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20251215 08:57:53.254607 20949 tablet_bootstrap.cc:492] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553: Bootstrap starting.
I20251215 08:57:53.255733 20950 raft_consensus.cc:359] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.256430 20950 raft_consensus.cc:385] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.256691 20950 raft_consensus.cc:740] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.257333 20950 consensus_queue.cc:260] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.259433 20950 ts_tablet_manager.cc:1434] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:57:53.261598 20949 tablet_bootstrap.cc:654] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553: Neither blocks nor log segments found. Creating new log.
I20251215 08:57:53.261499 20956 raft_consensus.cc:493] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.262048 20956 raft_consensus.cc:515] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.264104 20956 leader_election.cc:290] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.264542 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:53.265206 20758 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 0.
I20251215 08:57:53.266180 20798 leader_election.cc:304] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.266208 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:53.266887 20956 raft_consensus.cc:2804] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:53.267341 20956 raft_consensus.cc:493] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:53.266892 20908 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 0.
I20251215 08:57:53.267884 20956 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.273857 20949 tablet_bootstrap.cc:492] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553: No bootstrap required, opened a new log
I20251215 08:57:53.274322 20949 ts_tablet_manager.cc:1403] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s
I20251215 08:57:53.274063 20956 raft_consensus.cc:515] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.275828 20956 leader_election.cc:290] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.276835 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.276927 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.277323 20908 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.277515 20758 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.277348 20949 raft_consensus.cc:359] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.278115 20949 raft_consensus.cc:385] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:57:53.278425 20949 raft_consensus.cc:740] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Initialized, Role: FOLLOWER
I20251215 08:57:53.279084 20949 consensus_queue.cc:260] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.279939 20951 raft_consensus.cc:493] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.280395 20951 raft_consensus.cc:515] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.281416 20949 ts_tablet_manager.cc:1434] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251215 08:57:53.282434 20951 leader_election.cc:290] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.283952 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.284075 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:53.284420 20758 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 1.
I20251215 08:57:53.284590 20834 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 0.
I20251215 08:57:53.284756 20907 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 0.
I20251215 08:57:53.285439 20798 leader_election.cc:304] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.286142 20725 leader_election.cc:304] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:53.286403 20951 raft_consensus.cc:493] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.286880 20951 raft_consensus.cc:515] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.288733 20951 leader_election.cc:290] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.289151 20951 raft_consensus.cc:2804] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:53.289481 20951 raft_consensus.cc:493] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:53.289738 20951 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.297695 20908 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 1.
I20251215 08:57:53.298451 20956 raft_consensus.cc:2804] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.299579 20956 raft_consensus.cc:697] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Becoming Leader. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:57:53.300418 20956 consensus_queue.cc:237] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.301318 20951 raft_consensus.cc:515] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.302095 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.302594 20834 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 0.
I20251215 08:57:53.303256 20951 leader_election.cc:290] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.302318 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:53.304209 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.304337 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.304337 20725 leader_election.cc:304] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.304759 20834 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.305059 20907 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.305306 20951 raft_consensus.cc:2804] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:53.305681 20951 raft_consensus.cc:493] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:53.306047 20951 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.304744 20908 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 0.
I20251215 08:57:53.311789 20834 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 1.
I20251215 08:57:53.312258 20907 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 1.
I20251215 08:57:53.313331 20725 leader_election.cc:304] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:53.313344 20951 raft_consensus.cc:515] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.315390 20971 raft_consensus.cc:493] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.315127 20654 catalog_manager.cc:5654] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "6282ff6904b948e59e40f2db52b2be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.315956 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.315878 20971 raft_consensus.cc:515] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.316650 20834 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.321089 20833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576271add9c246529e6deb72528494f1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.322005 20833 raft_consensus.cc:2468] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
I20251215 08:57:53.323820 20834 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 1.
I20251215 08:57:53.324214 20874 leader_election.cc:304] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.325402 20955 raft_consensus.cc:2804] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:53.325785 21001 raft_consensus.cc:2804] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.326062 20955 raft_consensus.cc:493] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:53.326530 20955 raft_consensus.cc:3060] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.325404 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576271add9c246529e6deb72528494f1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:53.328800 20758 raft_consensus.cc:2468] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
I20251215 08:57:53.326570 20951 leader_election.cc:290] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.330029 20951 raft_consensus.cc:493] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.325489 20971 leader_election.cc:290] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.330632 20951 raft_consensus.cc:515] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.333294 20951 leader_election.cc:290] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.337594 21001 raft_consensus.cc:697] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:53.338593 21001 consensus_queue.cc:237] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.339113 20955 raft_consensus.cc:515] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.339414 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.340049 20907 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.340337 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.341114 20834 raft_consensus.cc:2393] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b0986965d0d49bf87a9aa87d2f06643 in current term 1: Already voted for candidate 48d193db12ff4e5b973328011a578553 in this term.
I20251215 08:57:53.341980 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576271add9c246529e6deb72528494f1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.342533 20758 raft_consensus.cc:3060] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.343329 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:53.345259 20725 leader_election.cc:304] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.346731 20907 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 1.
I20251215 08:57:53.347172 21003 raft_consensus.cc:2804] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.348042 21003 raft_consensus.cc:697] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:53.348805 20758 raft_consensus.cc:2468] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.349577 20725 leader_election.cc:304] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643; no voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:53.349859 20872 leader_election.cc:304] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:53.350513 21004 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.350699 20971 raft_consensus.cc:2804] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.351195 20971 raft_consensus.cc:697] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:53.352010 20971 consensus_queue.cc:237] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.352604 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576271add9c246529e6deb72528494f1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.353427 20834 raft_consensus.cc:3060] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.349175 21003 consensus_queue.cc:237] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.358510 21004 raft_consensus.cc:2749] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:53.365108 20955 leader_election.cc:290] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.365813 20834 raft_consensus.cc:2468] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.370884 20654 catalog_manager.cc:5654] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.371886 20971 raft_consensus.cc:493] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.372352 20971 raft_consensus.cc:515] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.372255 20656 catalog_manager.cc:5654] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 1 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:53.374081 20971 leader_election.cc:290] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.374843 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:53.375521 20758 raft_consensus.cc:2468] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
I20251215 08:57:53.375649 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.376540 20834 raft_consensus.cc:2468] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
I20251215 08:57:53.376525 20872 leader_election.cc:304] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:53.377568 20971 raft_consensus.cc:2804] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:53.377939 20971 raft_consensus.cc:493] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:53.378320 20971 raft_consensus.cc:3060] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.383945 20971 raft_consensus.cc:515] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.385536 20971 leader_election.cc:290] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.386394 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.386595 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.386982 20758 raft_consensus.cc:3060] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.387099 20834 raft_consensus.cc:3060] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.386965 20656 catalog_manager.cc:5654] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.392944 20758 raft_consensus.cc:2468] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.393554 20834 raft_consensus.cc:2468] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.394060 20872 leader_election.cc:304] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:53.394939 20971 raft_consensus.cc:2804] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.395411 20971 raft_consensus.cc:697] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:53.396227 20971 consensus_queue.cc:237] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.398515 20955 raft_consensus.cc:493] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.398978 20955 raft_consensus.cc:515] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.402346 20955 leader_election.cc:290] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.403482 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:53.403854 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.404271 20758 raft_consensus.cc:2393] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48d193db12ff4e5b973328011a578553 in current term 1: Already voted for candidate 6282ff6904b948e59e40f2db52b2be83 in this term.
I20251215 08:57:53.404047 20656 catalog_manager.cc:5654] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 1 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:53.405679 20872 leader_election.cc:304] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48d193db12ff4e5b973328011a578553; no voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:53.407737 20955 raft_consensus.cc:3060] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.414088 20955 raft_consensus.cc:2749] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:57:53.427253 21001 raft_consensus.cc:493] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.427771 21001 raft_consensus.cc:515] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.429570 21001 leader_election.cc:290] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.430465 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.430608 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553" is_pre_election: true
I20251215 08:57:53.431051 20834 raft_consensus.cc:2468] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 0.
I20251215 08:57:53.431149 20907 raft_consensus.cc:2468] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 0.
I20251215 08:57:53.432009 20725 leader_election.cc:304] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.432669 21001 raft_consensus.cc:2804] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:53.432941 21001 raft_consensus.cc:493] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:53.433267 21001 raft_consensus.cc:3060] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.438728 21001 raft_consensus.cc:515] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.440204 21001 leader_election.cc:290] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.441174 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.441324 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.441649 20834 raft_consensus.cc:3060] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.441813 20907 raft_consensus.cc:3060] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.446235 20834 raft_consensus.cc:2468] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 1.
I20251215 08:57:53.446519 20907 raft_consensus.cc:2468] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 1.
I20251215 08:57:53.447182 20725 leader_election.cc:304] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.447817 21001 raft_consensus.cc:2804] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.448173 21001 raft_consensus.cc:697] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:53.448918 21001 consensus_queue.cc:237] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.455535 20656 catalog_manager.cc:5654] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.472709 20956 consensus_queue.cc:1048] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.483094 20956 consensus_queue.cc:1048] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.494910 20955 raft_consensus.cc:493] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:57:53.495463 20955 raft_consensus.cc:515] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.497334 20955 leader_election.cc:290] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.498112 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643" is_pre_election: true
I20251215 08:57:53.498293 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83" is_pre_election: true
I20251215 08:57:53.498884 20834 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
I20251215 08:57:53.498821 20758 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 0.
I20251215 08:57:53.499835 20874 leader_election.cc:304] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.500749 20955 raft_consensus.cc:2804] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:57:53.501065 20955 raft_consensus.cc:493] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:57:53.501327 20955 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.506012 20955 raft_consensus.cc:515] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.507496 20955 leader_election.cc:290] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:53.508307 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.508555 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.508915 20758 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.509089 20834 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:57:53.510876 20955 consensus_queue.cc:1048] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.515801 20758 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.515801 20834 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 1.
I20251215 08:57:53.517251 20872 leader_election.cc:304] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:53.518054 20955 raft_consensus.cc:2804] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:57:53.518550 20955 raft_consensus.cc:697] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:53.519340 20955 consensus_queue.cc:237] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.523826 20971 consensus_queue.cc:1048] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.529839 20656 catalog_manager.cc:5654] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 1 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:53.569463 20971 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.579190 20975 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:57:53.643450 20907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74"
dest_uuid: "48d193db12ff4e5b973328011a578553"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:43108
I20251215 08:57:53.644021 20907 raft_consensus.cc:606] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:57:53.648594 20907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dc67dabaa62248d28af779466ba95976"
dest_uuid: "48d193db12ff4e5b973328011a578553"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:43108
I20251215 08:57:53.649120 20907 raft_consensus.cc:606] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:57:53.665844 20834 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a34845828f9a42118a523af87e308b08"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:45808
I20251215 08:57:53.666352 20834 raft_consensus.cc:606] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:57:53.671164 20907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a058316801f041a898362d1fdd5db080"
dest_uuid: "48d193db12ff4e5b973328011a578553"
mode: GRACEFUL
new_leader_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:43108
I20251215 08:57:53.671660 20907 raft_consensus.cc:606] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Received request to transfer leadership to TS 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:53.685626 20759 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
mode: GRACEFUL
new_leader_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:40130
I20251215 08:57:53.686098 20759 raft_consensus.cc:606] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Received request to transfer leadership to TS 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:53.690712 20908 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1"
dest_uuid: "48d193db12ff4e5b973328011a578553"
mode: GRACEFUL
new_leader_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:43108
I20251215 08:57:53.691262 20908 raft_consensus.cc:606] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Received request to transfer leadership to TS 6282ff6904b948e59e40f2db52b2be83
I20251215 08:57:53.691457 20974 consensus_queue.cc:1048] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.701282 20978 consensus_queue.cc:1048] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:57:53.710271 20956 raft_consensus.cc:993] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:57:53.710759 20978 raft_consensus.cc:1081] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:57:53.712286 20759 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a34845828f9a42118a523af87e308b08"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40114
I20251215 08:57:53.712869 20759 raft_consensus.cc:493] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:53.713313 20759 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.717881 20759 raft_consensus.cc:515] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.719600 20759 leader_election.cc:290] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.720816 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.721428 20908 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.721410 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:53.722160 20834 raft_consensus.cc:3055] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:53.722476 20834 raft_consensus.cc:740] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:57:53.723237 20834 consensus_queue.cc:260] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.724402 20834 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.728474 20908 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:57:53.729513 20725 leader_election.cc:304] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:53.730170 21001 raft_consensus.cc:2804] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:53.730651 21001 raft_consensus.cc:697] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:53.730932 20834 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:57:53.731470 21001 consensus_queue.cc:237] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.736902 20654 catalog_manager.cc:5654] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 1 to 2, leader changed from 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.743120 21001 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.752877 20951 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.802793 20975 consensus_queue.cc:1048] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.812238 20976 consensus_queue.cc:1048] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.815254 20955 consensus_queue.cc:1048] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.827824 20975 consensus_queue.cc:1048] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.836360 20971 raft_consensus.cc:993] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553: : Instructing follower 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:57:53.836865 20971 raft_consensus.cc:1081] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Signalling peer 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:57:53.838325 20834 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:45802
I20251215 08:57:53.838769 20834 raft_consensus.cc:493] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:53.839052 20834 raft_consensus.cc:3060] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.843813 20834 raft_consensus.cc:515] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.845343 20834 leader_election.cc:290] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.846215 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.846344 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6807ea1cd51f4491aaa320e8382f23e1" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.846892 20759 raft_consensus.cc:3060] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.846920 20908 raft_consensus.cc:3055] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:53.847431 20908 raft_consensus.cc:740] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:53.848237 20908 consensus_queue.cc:260] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.849516 20908 raft_consensus.cc:3060] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.853870 20759 raft_consensus.cc:2468] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:57:53.855141 20798 leader_election.cc:304] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.855927 20978 raft_consensus.cc:2804] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:53.856297 20908 raft_consensus.cc:2468] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:57:53.856318 20978 raft_consensus.cc:697] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Becoming Leader. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:57:53.857421 20978 consensus_queue.cc:237] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.863219 20654 catalog_manager.cc:5654] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 reported cstate change: term changed from 1 to 2, leader changed from 48d193db12ff4e5b973328011a578553 (127.19.163.67) to 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "6282ff6904b948e59e40f2db52b2be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.879383 20951 consensus_queue.cc:1048] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.889732 21004 consensus_queue.cc:1048] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.901844 21004 consensus_queue.cc:1048] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.911933 20951 consensus_queue.cc:1048] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:57:53.921279 21004 raft_consensus.cc:993] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643: : Instructing follower 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:57:53.921696 20951 raft_consensus.cc:1081] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Signalling peer 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:57:53.923131 20834 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:45798
I20251215 08:57:53.923777 20834 raft_consensus.cc:493] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:53.924211 20834 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.930011 20834 raft_consensus.cc:515] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.931392 20834 leader_election.cc:290] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.932367 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.932524 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.932951 20759 raft_consensus.cc:3055] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:53.933079 20908 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.933341 20759 raft_consensus.cc:740] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:53.933966 20759 consensus_queue.cc:260] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.934868 20759 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.939265 20908 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:57:53.939827 20759 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:57:53.940311 20800 leader_election.cc:304] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.941044 20978 raft_consensus.cc:2804] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:53.941426 20978 raft_consensus.cc:697] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Becoming Leader. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:57:53.942173 20978 consensus_queue.cc:237] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.947909 20655 catalog_manager.cc:5654] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 reported cstate change: term changed from 1 to 2, leader changed from 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65) to 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "6282ff6904b948e59e40f2db52b2be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:53.953491 20975 consensus_queue.cc:1048] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.965044 20976 consensus_queue.cc:1048] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:57:53.972810 20975 raft_consensus.cc:993] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553: : Instructing follower 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:57:53.973320 20976 raft_consensus.cc:1081] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Signalling peer 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:57:53.974844 20834 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a058316801f041a898362d1fdd5db080"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:45802
I20251215 08:57:53.975346 20834 raft_consensus.cc:493] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:53.975623 20834 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.980261 20834 raft_consensus.cc:515] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.982259 20834 leader_election.cc:290] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:53.982676 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:53.982884 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:53.983278 20759 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.983410 20908 raft_consensus.cc:3055] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:53.983731 20908 raft_consensus.cc:740] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:53.984361 20908 consensus_queue.cc:260] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.985242 20908 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:53.988965 20759 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:57:53.989924 20798 leader_election.cc:304] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:53.990149 20908 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:57:53.990568 20978 raft_consensus.cc:2804] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:53.990968 20978 raft_consensus.cc:697] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Becoming Leader. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:57:53.991637 20978 consensus_queue.cc:237] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:53.997697 20654 catalog_manager.cc:5654] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 reported cstate change: term changed from 1 to 2, leader changed from 48d193db12ff4e5b973328011a578553 (127.19.163.67) to 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "6282ff6904b948e59e40f2db52b2be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:54.001374 20975 raft_consensus.cc:993] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:57:54.001821 20976 raft_consensus.cc:1081] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:57:54.003182 20759 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40112
I20251215 08:57:54.003651 20759 raft_consensus.cc:493] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:54.004029 20759 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:54.008948 20759 raft_consensus.cc:515] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:54.010486 20759 leader_election.cc:290] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:54.011246 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:54.011539 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:54.011807 20834 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:54.012056 20908 raft_consensus.cc:3055] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:54.012329 20908 raft_consensus.cc:740] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:54.012959 20908 consensus_queue.cc:260] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:54.013875 20908 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:54.016525 20834 raft_consensus.cc:2468] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:57:54.017553 20725 leader_election.cc:304] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:54.018215 20951 raft_consensus.cc:2804] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:54.018623 20951 raft_consensus.cc:697] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:54.018806 20908 raft_consensus.cc:2468] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:57:54.019436 20951 consensus_queue.cc:237] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:54.025732 20654 catalog_manager.cc:5654] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 1 to 2, leader changed from 48d193db12ff4e5b973328011a578553 (127.19.163.67) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:54.111713 20975 raft_consensus.cc:993] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:57:54.112145 20976 raft_consensus.cc:1081] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:57:54.113552 20759 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dc67dabaa62248d28af779466ba95976"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40112
I20251215 08:57:54.114068 20759 raft_consensus.cc:493] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:54.114362 20759 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:54.119305 20759 raft_consensus.cc:515] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:54.120980 20759 leader_election.cc:290] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:57:54.121701 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:54.122084 20908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:57:54.122187 20834 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:54.122561 20908 raft_consensus.cc:3055] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:54.122815 20908 raft_consensus.cc:740] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:54.123436 20908 consensus_queue.cc:260] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:54.124315 20908 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:54.126641 20834 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:57:54.127535 20725 leader_election.cc:304] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:54.128170 20951 raft_consensus.cc:2804] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:54.128628 20951 raft_consensus.cc:697] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:54.129276 20908 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:57:54.129537 20951 consensus_queue.cc:237] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:54.135802 20655 catalog_manager.cc:5654] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 1 to 2, leader changed from 48d193db12ff4e5b973328011a578553 (127.19.163.67) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:57:54.203233 20834 raft_consensus.cc:1275] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.204226 20951 consensus_queue.cc:1048] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.211191 20908 raft_consensus.cc:1275] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.212677 20951 consensus_queue.cc:1048] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.357549 20759 raft_consensus.cc:1275] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.358697 20956 consensus_queue.cc:1048] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.366770 20908 raft_consensus.cc:1275] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.368124 20978 consensus_queue.cc:1048] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.402410 20908 raft_consensus.cc:1275] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.403611 20978 consensus_queue.cc:1048] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.411145 20759 raft_consensus.cc:1275] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.412542 20974 consensus_queue.cc:1048] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.525597 20759 raft_consensus.cc:1275] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.526811 20974 consensus_queue.cc:1048] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.535069 20908 raft_consensus.cc:1275] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.537304 20978 consensus_queue.cc:1048] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:57:54.546223 20907 raft_consensus.cc:1275] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.547623 21001 consensus_queue.cc:1048] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.555639 20834 raft_consensus.cc:1275] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.557340 21003 consensus_queue.cc:1048] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.566643 20907 raft_consensus.cc:1275] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.567831 21004 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:54.576771 20834 raft_consensus.cc:1275] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:54.578128 20951 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:56.723439 20109 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20251215 08:57:56.724048 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:56.739614 20759 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1b8468317e1e402aba91c174231624e2"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
mode: GRACEFUL
new_leader_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:53430
I20251215 08:57:56.740092 20759 raft_consensus.cc:606] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Received request to transfer leadership to TS 48d193db12ff4e5b973328011a578553
I20251215 08:57:56.742311 20759 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dc67dabaa62248d28af779466ba95976"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
mode: GRACEFUL
new_leader_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:53430
I20251215 08:57:56.742759 20759 raft_consensus.cc:606] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Received request to transfer leadership to TS 48d193db12ff4e5b973328011a578553
I20251215 08:57:56.743749 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251215 08:57:56.744156 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:56.998993 20951 raft_consensus.cc:993] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: : Instructing follower 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:57:56.999464 21043 raft_consensus.cc:1081] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Signalling peer 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:57:57.000768 20907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1b8468317e1e402aba91c174231624e2"
dest_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:43084
I20251215 08:57:57.001207 20907 raft_consensus.cc:493] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:57.001466 20907 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:57.006122 20907 raft_consensus.cc:515] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:57.007603 20907 leader_election.cc:290] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 2 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:57.008600 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:57.008647 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:57.009305 20759 raft_consensus.cc:3055] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:57:57.009341 20834 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:57.009600 20759 raft_consensus.cc:740] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:57.010257 20759 consensus_queue.cc:260] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:57.011145 20759 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:57:57.013620 20834 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 2.
I20251215 08:57:57.014575 20874 leader_election.cc:304] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:57:57.015225 20975 raft_consensus.cc:2804] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:57:57.015863 20759 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 2.
I20251215 08:57:57.016204 20975 raft_consensus.cc:697] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:57.017120 20975 consensus_queue.cc:237] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:57.023968 20655 catalog_manager.cc:5654] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 1 to 2, leader changed from 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65) to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 2 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:57.056510 20951 raft_consensus.cc:993] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643: : Instructing follower 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:57:57.056932 20951 raft_consensus.cc:1081] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Signalling peer 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:57:57.058277 20907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dc67dabaa62248d28af779466ba95976"
dest_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:43084
I20251215 08:57:57.058735 20907 raft_consensus.cc:493] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:57:57.059006 20907 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:57.063618 20907 raft_consensus.cc:515] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:57.065100 20907 leader_election.cc:290] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 3 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:57:57.065984 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:57:57.066085 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:57:57.066439 20759 raft_consensus.cc:3055] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:57:57.066555 20834 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:57.066702 20759 raft_consensus.cc:740] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:57:57.067322 20759 consensus_queue.cc:260] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:57.068117 20759 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:57:57.072265 20759 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 3.
I20251215 08:57:57.072630 20834 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 3.
I20251215 08:57:57.073133 20872 leader_election.cc:304] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:57:57.073918 20975 raft_consensus.cc:2804] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:57:57.074388 20975 raft_consensus.cc:697] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:57:57.075189 20975 consensus_queue.cc:237] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:57:57.083253 20655 catalog_manager.cc:5654] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 2 to 3, leader changed from 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65) to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 3 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:57:57.531647 20834 raft_consensus.cc:1275] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:57.532684 20975 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:57.538136 20759 raft_consensus.cc:1275] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:57.540576 21046 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:57:57.550129 20758 raft_consensus.cc:1275] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:57:57.552439 21054 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:57:57.560405 20834 raft_consensus.cc:1275] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:57:57.561851 20975 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:57:57.745056 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:57.750415 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:57.750856 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:58.751816 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:58.757211 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:58.757659 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:57:59.758652 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:57:59.763844 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:57:59.764251 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:00.765118 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:00.770373 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:00.770756 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:01.771867 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:01.777788 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:01.778234 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:02.779214 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:02.784440 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:02.784852 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:03.785787 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:03.790942 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:03.791363 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:04.792224 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:04.797564 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:04.798038 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:05.798978 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:05.804070 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:05.804509 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:07.025851 20109 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20251215 08:58:07.032310 20907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dc67dabaa62248d28af779466ba95976"
dest_uuid: "48d193db12ff4e5b973328011a578553"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:43108
I20251215 08:58:07.032930 20907 raft_consensus.cc:606] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:58:07.037094 20907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1b8468317e1e402aba91c174231624e2"
dest_uuid: "48d193db12ff4e5b973328011a578553"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:43108
I20251215 08:58:07.037492 20907 raft_consensus.cc:606] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:58:07.046100 20833 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a058316801f041a898362d1fdd5db080"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:45808
I20251215 08:58:07.046651 20833 raft_consensus.cc:606] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:58:07.051084 20833 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fd7013ed60464deca91ae876affc032d"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:45808
I20251215 08:58:07.051632 20833 raft_consensus.cc:606] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:58:07.056560 20834 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
mode: GRACEFUL
new_leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:45808
I20251215 08:58:07.056980 20834 raft_consensus.cc:606] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Received request to transfer leadership to TS 2b0986965d0d49bf87a9aa87d2f06643
I20251215 08:58:07.078555 21075 raft_consensus.cc:993] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.079092 21069 raft_consensus.cc:1081] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.080523 20757 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1b8468317e1e402aba91c174231624e2"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40112
I20251215 08:58:07.080963 20757 raft_consensus.cc:493] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:07.081270 20757 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.086812 20757 raft_consensus.cc:515] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.088379 20757 leader_election.cc:290] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 3 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:58:07.089322 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:58:07.089614 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:58:07.090040 20834 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.090093 20907 raft_consensus.cc:3055] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:58:07.090467 20907 raft_consensus.cc:740] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:58:07.091315 20907 consensus_queue.cc:260] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.092610 20907 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.097281 20834 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 3.
I20251215 08:58:07.098557 20725 leader_election.cc:304] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:58:07.099311 20907 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 3.
I20251215 08:58:07.099511 21074 raft_consensus.cc:2804] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:58:07.100032 21074 raft_consensus.cc:697] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:07.100800 21074 consensus_queue.cc:237] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.110276 20654 catalog_manager.cc:5654] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 2 to 3, leader changed from 48d193db12ff4e5b973328011a578553 (127.19.163.67) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 3 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:07.158143 21068 raft_consensus.cc:993] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.158668 21072 raft_consensus.cc:1081] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.160163 20757 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fd7013ed60464deca91ae876affc032d"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40114
I20251215 08:58:07.160612 20757 raft_consensus.cc:493] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:07.160873 20757 raft_consensus.cc:3060] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:07.165755 20757 raft_consensus.cc:515] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.167210 20757 leader_election.cc:290] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:58:07.168167 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:58:07.168498 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd7013ed60464deca91ae876affc032d" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:58:07.168735 20834 raft_consensus.cc:3055] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:07.169045 20907 raft_consensus.cc:3060] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:07.169057 20834 raft_consensus.cc:740] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:58:07.169945 20834 consensus_queue.cc:260] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.171540 20834 raft_consensus.cc:3060] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:07.178598 20834 raft_consensus.cc:2468] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:58:07.179838 20725 leader_election.cc:304] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:58:07.175487 20907 raft_consensus.cc:2468] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 2.
I20251215 08:58:07.180760 21074 raft_consensus.cc:2804] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:07.181259 21074 raft_consensus.cc:697] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:07.182166 21074 consensus_queue.cc:237] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.184661 21069 raft_consensus.cc:993] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.185086 21075 raft_consensus.cc:1081] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.186407 20757 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dc67dabaa62248d28af779466ba95976"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40112
I20251215 08:58:07.186910 20757 raft_consensus.cc:493] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:07.187211 20757 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:58:07.193539 20757 raft_consensus.cc:515] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.194952 21068 raft_consensus.cc:993] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.195433 21072 raft_consensus.cc:1081] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.195950 20757 leader_election.cc:290] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 4 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:58:07.195626 20654 catalog_manager.cc:5654] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 1 to 2, leader changed from 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:07.196520 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:58:07.196990 20834 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:58:07.197242 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc67dabaa62248d28af779466ba95976" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:58:07.197746 20907 raft_consensus.cc:3055] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:58:07.197664 20757 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40114
I20251215 08:58:07.198093 20907 raft_consensus.cc:740] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:58:07.198487 20757 raft_consensus.cc:493] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:07.198908 20907 consensus_queue.cc:260] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.198887 20757 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.200111 20907 raft_consensus.cc:3060] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:58:07.202785 20834 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 4.
I20251215 08:58:07.203608 20725 leader_election.cc:304] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:58:07.204200 21074 raft_consensus.cc:2804] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:58:07.204730 21074 raft_consensus.cc:697] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:07.205659 21074 consensus_queue.cc:237] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.208665 20907 raft_consensus.cc:2468] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 4.
I20251215 08:58:07.208958 20757 raft_consensus.cc:515] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.211951 20757 leader_election.cc:290] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 3 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:58:07.212632 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:58:07.212667 20654 catalog_manager.cc:5654] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 3 to 4, leader changed from 48d193db12ff4e5b973328011a578553 (127.19.163.67) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 4 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:07.213299 20834 raft_consensus.cc:3055] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:58:07.213371 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "691a85c9bd184f6788b51c19d2ad7273" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:58:07.213798 20834 raft_consensus.cc:740] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:58:07.213953 20907 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.214570 20834 consensus_queue.cc:260] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.215829 20834 raft_consensus.cc:3060] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.221319 20907 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 3.
I20251215 08:58:07.222489 20725 leader_election.cc:304] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:58:07.223074 21074 raft_consensus.cc:2804] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:58:07.223441 21074 raft_consensus.cc:697] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:07.223469 20834 raft_consensus.cc:2468] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 3.
I20251215 08:58:07.224407 21074 consensus_queue.cc:237] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.231500 20656 catalog_manager.cc:5654] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 2 to 3, leader changed from 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 3 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:07.533710 20907 raft_consensus.cc:1275] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:07.534700 21074 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:07.544620 20834 raft_consensus.cc:1275] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:07.545887 21074 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:07.553422 21068 raft_consensus.cc:993] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83: : Instructing follower 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.553941 21072 raft_consensus.cc:1081] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Signalling peer 2b0986965d0d49bf87a9aa87d2f06643 to start an election
I20251215 08:58:07.555684 20757 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a058316801f041a898362d1fdd5db080"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
from {username='slave'} at 127.0.0.1:40114
I20251215 08:58:07.556398 20757 raft_consensus.cc:493] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:07.556746 20757 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.562348 20757 raft_consensus.cc:515] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.563944 20757 leader_election.cc:290] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 3 election: Requested vote from peers 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:58:07.564675 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:58:07.564944 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a058316801f041a898362d1fdd5db080" candidate_uuid: "2b0986965d0d49bf87a9aa87d2f06643" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:58:07.565238 20834 raft_consensus.cc:3055] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:58:07.565533 20907 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.565543 20834 raft_consensus.cc:740] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:58:07.566463 20834 consensus_queue.cc:260] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.567492 20834 raft_consensus.cc:3060] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:07.570407 20907 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 3.
I20251215 08:58:07.571309 20725 leader_election.cc:304] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:58:07.571622 20834 raft_consensus.cc:2468] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0986965d0d49bf87a9aa87d2f06643 in term 3.
I20251215 08:58:07.571966 21074 raft_consensus.cc:2804] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:58:07.572444 21074 raft_consensus.cc:697] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Becoming Leader. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:07.573299 21074 consensus_queue.cc:237] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:07.581202 20834 raft_consensus.cc:1275] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:07.580952 20656 catalog_manager.cc:5654] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 reported cstate change: term changed from 2 to 3, leader changed from 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66) to 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65). New cstate: current_term: 3 leader_uuid: "2b0986965d0d49bf87a9aa87d2f06643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:07.582433 21074 consensus_queue.cc:1048] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:07.590885 20907 raft_consensus.cc:1275] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:07.592255 21074 consensus_queue.cc:1048] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:07.613734 20834 raft_consensus.cc:1275] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:58:07.614799 21074 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:07.624872 20907 raft_consensus.cc:1275] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 4 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:58:07.626238 20951 consensus_queue.cc:1048] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251215 08:58:07.700448 20855 debug-util.cc:398] Leaking SignalData structure 0x7b08001048c0 after lost signal to thread 20112
W20251215 08:58:07.701403 20855 debug-util.cc:398] Leaking SignalData structure 0x7b0800104fa0 after lost signal to thread 20691
W20251215 08:58:07.702234 20855 debug-util.cc:398] Leaking SignalData structure 0x7b0800089800 after lost signal to thread 20783
W20251215 08:58:07.702994 20855 debug-util.cc:398] Leaking SignalData structure 0x7b0800114a80 after lost signal to thread 20858
W20251215 08:58:07.703696 20855 debug-util.cc:398] Leaking SignalData structure 0x7b08001a07e0 after lost signal to thread 20932
I20251215 08:58:07.816090 20834 raft_consensus.cc:1275] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:07.817440 21087 consensus_queue.cc:1048] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:07.829938 20907 raft_consensus.cc:1275] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:07.833755 21074 consensus_queue.cc:1048] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:08.083352 20907 raft_consensus.cc:1275] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:08.084529 20951 consensus_queue.cc:1048] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:08.091189 20834 raft_consensus.cc:1275] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Refusing update from remote peer 2b0986965d0d49bf87a9aa87d2f06643: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:08.093739 20951 consensus_queue.cc:1048] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:10.092084 20109 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20251215 08:58:10.092905 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:10.104197 20757 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1b8468317e1e402aba91c174231624e2"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
mode: GRACEFUL
new_leader_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:53430
I20251215 08:58:10.104676 20757 raft_consensus.cc:606] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Received request to transfer leadership to TS 48d193db12ff4e5b973328011a578553
I20251215 08:58:10.107295 20757 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a34845828f9a42118a523af87e308b08"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
mode: GRACEFUL
new_leader_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:53430
I20251215 08:58:10.107808 20757 raft_consensus.cc:606] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Received request to transfer leadership to TS 48d193db12ff4e5b973328011a578553
I20251215 08:58:10.110152 20757 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
mode: GRACEFUL
new_leader_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:53430
I20251215 08:58:10.110600 20757 raft_consensus.cc:606] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Received request to transfer leadership to TS 6282ff6904b948e59e40f2db52b2be83
I20251215 08:58:10.113095 20757 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807"
dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
mode: GRACEFUL
new_leader_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:53430
I20251215 08:58:10.113575 20757 raft_consensus.cc:606] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Received request to transfer leadership to TS 6282ff6904b948e59e40f2db52b2be83
I20251215 08:58:10.114492 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251215 08:58:10.114996 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:10.118592 21100 raft_consensus.cc:993] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: : Instructing follower 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:58:10.119045 21087 raft_consensus.cc:1081] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Signalling peer 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:58:10.120771 20908 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1b8468317e1e402aba91c174231624e2"
dest_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:43084
I20251215 08:58:10.123075 20908 raft_consensus.cc:493] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:10.123438 20908 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:58:10.125442 21087 raft_consensus.cc:993] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643: : Instructing follower 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:58:10.125900 21087 raft_consensus.cc:1081] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Signalling peer 48d193db12ff4e5b973328011a578553 to start an election
I20251215 08:58:10.130406 20908 raft_consensus.cc:515] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.131791 20907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a34845828f9a42118a523af87e308b08"
dest_uuid: "48d193db12ff4e5b973328011a578553"
from {username='slave'} at 127.0.0.1:43084
I20251215 08:58:10.132325 20907 raft_consensus.cc:493] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:10.132716 20907 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:10.133222 20834 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:58:10.133162 20757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b8468317e1e402aba91c174231624e2" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:58:10.133780 20834 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:58:10.133822 20757 raft_consensus.cc:3055] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:58:10.134200 20757 raft_consensus.cc:740] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:10.134869 20757 consensus_queue.cc:260] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.135874 20757 raft_consensus.cc:3060] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:58:10.140306 20907 raft_consensus.cc:515] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.141980 20757 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 4.
I20251215 08:58:10.143491 20872 leader_election.cc:304] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0986965d0d49bf87a9aa87d2f06643, 48d193db12ff4e5b973328011a578553; no voters:
I20251215 08:58:10.144239 21075 raft_consensus.cc:2804] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:58:10.146166 20833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
I20251215 08:58:10.146796 20833 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:10.143482 20834 raft_consensus.cc:2468] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 4.
I20251215 08:58:10.148281 20908 leader_election.cc:290] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 4 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:58:10.149578 20757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34845828f9a42118a523af87e308b08" candidate_uuid: "48d193db12ff4e5b973328011a578553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:58:10.150244 20757 raft_consensus.cc:3055] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:58:10.150609 20757 raft_consensus.cc:740] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:10.151525 20757 consensus_queue.cc:260] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.152922 20757 raft_consensus.cc:3060] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:10.157024 20833 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 3.
I20251215 08:58:10.158139 20907 leader_election.cc:290] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 3 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293)
I20251215 08:58:10.159323 20874 leader_election.cc:304] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:58:10.159726 21075 raft_consensus.cc:697] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 4 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:58:10.160735 21102 raft_consensus.cc:2804] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:58:10.160634 21075 consensus_queue.cc:237] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.161289 21102 raft_consensus.cc:697] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 3 LEADER]: Becoming Leader. State: Replica: 48d193db12ff4e5b973328011a578553, State: Running, Role: LEADER
I20251215 08:58:10.162216 21102 consensus_queue.cc:237] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.167814 20757 raft_consensus.cc:2468] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48d193db12ff4e5b973328011a578553 in term 3.
I20251215 08:58:10.178195 20655 catalog_manager.cc:5654] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 3 to 4, leader changed from 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65) to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 4 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:10.195201 20655 catalog_manager.cc:5654] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 reported cstate change: term changed from 2 to 3, leader changed from 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65) to 48d193db12ff4e5b973328011a578553 (127.19.163.67). New cstate: current_term: 3 leader_uuid: "48d193db12ff4e5b973328011a578553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:10.307258 21100 raft_consensus.cc:993] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643: : Instructing follower 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:58:10.307757 21087 raft_consensus.cc:1081] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Signalling peer 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:58:10.309060 20833 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:45798
I20251215 08:58:10.309516 20833 raft_consensus.cc:493] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:10.309787 20833 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:10.314476 20833 raft_consensus.cc:515] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.316584 20833 leader_election.cc:290] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 3 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:58:10.317409 20757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:58:10.317708 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd4e1cf18544f84a1a61cb7df061b74" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:58:10.317981 20757 raft_consensus.cc:3055] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:58:10.318223 20907 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:10.318315 20757 raft_consensus.cc:740] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:10.319135 20757 consensus_queue.cc:260] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.320194 20757 raft_consensus.cc:3060] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:10.324542 20907 raft_consensus.cc:2468] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 3.
I20251215 08:58:10.325788 20800 leader_election.cc:304] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:58:10.326202 20757 raft_consensus.cc:2468] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 3.
I20251215 08:58:10.326473 21072 raft_consensus.cc:2804] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:58:10.327709 21072 raft_consensus.cc:697] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 3 LEADER]: Becoming Leader. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:58:10.328474 21072 consensus_queue.cc:237] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.335624 20655 catalog_manager.cc:5654] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 reported cstate change: term changed from 2 to 3, leader changed from 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65) to 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66). New cstate: current_term: 3 leader_uuid: "6282ff6904b948e59e40f2db52b2be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:10.466418 21100 raft_consensus.cc:993] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643: : Instructing follower 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:58:10.466797 21087 raft_consensus.cc:1081] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Signalling peer 6282ff6904b948e59e40f2db52b2be83 to start an election
I20251215 08:58:10.467880 20833 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807"
dest_uuid: "6282ff6904b948e59e40f2db52b2be83"
from {username='slave'} at 127.0.0.1:45798
I20251215 08:58:10.468297 20833 raft_consensus.cc:493] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:10.468569 20833 raft_consensus.cc:3060] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:10.473137 20833 raft_consensus.cc:515] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.474525 20833 leader_election.cc:290] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Requested vote from peers 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261), 48d193db12ff4e5b973328011a578553 (127.19.163.67:41687)
I20251215 08:58:10.475337 20757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2b0986965d0d49bf87a9aa87d2f06643"
I20251215 08:58:10.475436 20907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaea97f463c4731b8e73ee3d0028807" candidate_uuid: "6282ff6904b948e59e40f2db52b2be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48d193db12ff4e5b973328011a578553"
I20251215 08:58:10.475762 20757 raft_consensus.cc:3055] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:10.475888 20907 raft_consensus.cc:3060] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:10.476016 20757 raft_consensus.cc:740] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b0986965d0d49bf87a9aa87d2f06643, State: Running, Role: LEADER
I20251215 08:58:10.476724 20757 consensus_queue.cc:260] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.477587 20757 raft_consensus.cc:3060] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:10.480703 20907 raft_consensus.cc:2468] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:58:10.481688 20800 leader_election.cc:304] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d193db12ff4e5b973328011a578553, 6282ff6904b948e59e40f2db52b2be83; no voters:
I20251215 08:58:10.482110 20757 raft_consensus.cc:2468] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6282ff6904b948e59e40f2db52b2be83 in term 2.
I20251215 08:58:10.482498 21110 raft_consensus.cc:2804] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:10.483180 21110 raft_consensus.cc:697] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Becoming Leader. State: Replica: 6282ff6904b948e59e40f2db52b2be83, State: Running, Role: LEADER
I20251215 08:58:10.484087 21110 consensus_queue.cc:237] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } }
I20251215 08:58:10.490474 20655 catalog_manager.cc:5654] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 reported cstate change: term changed from 1 to 2, leader changed from 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65) to 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "6282ff6904b948e59e40f2db52b2be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:10.552347 20757 raft_consensus.cc:1275] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:10.552712 20833 raft_consensus.cc:1275] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:10.553557 21102 consensus_queue.cc:1048] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:10.554181 21075 consensus_queue.cc:1048] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:10.602958 20757 raft_consensus.cc:1275] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:58:10.604074 21075 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:10.612226 20834 raft_consensus.cc:1275] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Refusing update from remote peer 48d193db12ff4e5b973328011a578553: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:58:10.613538 21075 consensus_queue.cc:1048] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6282ff6904b948e59e40f2db52b2be83" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:10.816439 20757 raft_consensus.cc:1275] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:10.817631 21110 consensus_queue.cc:1048] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:10.825663 20907 raft_consensus.cc:1275] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:58:10.827606 21121 consensus_queue.cc:1048] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:10.867815 20757 raft_consensus.cc:1275] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:10.868885 21121 consensus_queue.cc:1048] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0986965d0d49bf87a9aa87d2f06643" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:10.877067 20907 raft_consensus.cc:1275] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Refusing update from remote peer 6282ff6904b948e59e40f2db52b2be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:10.878746 21121 consensus_queue.cc:1048] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d193db12ff4e5b973328011a578553" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:11.115916 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:11.121526 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:11.121925 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:12.123054 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:12.130334 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:12.130822 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:13.131875 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:13.138046 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:13.138574 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:14.139525 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:14.144729 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:14.145118 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:15.145965 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:15.150952 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:15.151326 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:16.152194 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:16.157874 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:16.158300 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:17.159171 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:17.165228 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:17.165771 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:18.166728 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:18.173471 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:18.173866 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:19.174695 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:19.179893 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251215 08:58:19.180311 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:20.210315 20109 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20251215 08:58:20.210903 20109 tablet_server.cc:178] TabletServer@127.19.163.65:0 shutting down...
I20251215 08:58:20.241842 20109 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 08:58:20.242604 20109 tablet_replica.cc:333] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.243271 20109 raft_consensus.cc:2243] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.243947 20109 raft_consensus.cc:2272] T 6807ea1cd51f4491aaa320e8382f23e1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.245867 20109 tablet_replica.cc:333] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.246403 20109 raft_consensus.cc:2243] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.247179 20109 raft_consensus.cc:2272] T 691a85c9bd184f6788b51c19d2ad7273 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.249095 20109 tablet_replica.cc:333] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.249557 20109 raft_consensus.cc:2243] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.249929 20109 raft_consensus.cc:2272] T 576271add9c246529e6deb72528494f1 P 2b0986965d0d49bf87a9aa87d2f06643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.251529 20109 tablet_replica.cc:333] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.252246 20109 raft_consensus.cc:2243] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.253113 20109 raft_consensus.cc:2272] T dc67dabaa62248d28af779466ba95976 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.255276 20109 tablet_replica.cc:333] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.255862 20109 raft_consensus.cc:2243] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.256295 20109 raft_consensus.cc:2272] T a34845828f9a42118a523af87e308b08 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.258301 20109 tablet_replica.cc:333] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.258908 20109 raft_consensus.cc:2243] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.259490 20109 raft_consensus.cc:2272] T 1b8468317e1e402aba91c174231624e2 P 2b0986965d0d49bf87a9aa87d2f06643 [term 4 FOLLOWER]: Raft consensus is shut down!
W20251215 08:58:20.261718 20872 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20251215 08:58:20.262096 20109 tablet_replica.cc:333] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.262660 20109 raft_consensus.cc:2243] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.263617 20109 raft_consensus.cc:2272] T a058316801f041a898362d1fdd5db080 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Raft consensus is shut down!
W20251215 08:58:20.265081 20872 consensus_peers.cc:597] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 -> Peer 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261): Couldn't send request to peer 2b0986965d0d49bf87a9aa87d2f06643. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:20.266134 20109 tablet_replica.cc:333] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.266630 20109 raft_consensus.cc:2243] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.267345 20109 raft_consensus.cc:2272] T fd7013ed60464deca91ae876affc032d P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.269061 20109 tablet_replica.cc:333] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.269500 20109 raft_consensus.cc:2243] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.269896 20109 raft_consensus.cc:2272] T ccaea97f463c4731b8e73ee3d0028807 P 2b0986965d0d49bf87a9aa87d2f06643 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.271587 20109 tablet_replica.cc:333] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643: stopping tablet replica
I20251215 08:58:20.271991 20109 raft_consensus.cc:2243] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.272352 20109 raft_consensus.cc:2272] T afd4e1cf18544f84a1a61cb7df061b74 P 2b0986965d0d49bf87a9aa87d2f06643 [term 3 FOLLOWER]: Raft consensus is shut down!
W20251215 08:58:20.293995 20798 consensus_peers.cc:597] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 -> Peer 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261): Couldn't send request to peer 2b0986965d0d49bf87a9aa87d2f06643. Status: Network error: Client connection negotiation failed: client connection to 127.19.163.65:43261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:20.295053 20109 tablet_server.cc:195] TabletServer@127.19.163.65:0 shutdown complete.
W20251215 08:58:20.296595 20798 consensus_peers.cc:597] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 -> Peer 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261): Couldn't send request to peer 2b0986965d0d49bf87a9aa87d2f06643. Status: Network error: Client connection negotiation failed: client connection to 127.19.163.65:43261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:20.314512 20109 tablet_server.cc:178] TabletServer@127.19.163.66:0 shutting down...
I20251215 08:58:20.343909 20109 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 08:58:20.344575 20109 tablet_replica.cc:333] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.345198 20109 raft_consensus.cc:2243] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.345971 20109 raft_consensus.cc:2272] T 6807ea1cd51f4491aaa320e8382f23e1 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.347882 20109 tablet_replica.cc:333] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.348371 20109 raft_consensus.cc:2243] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.348769 20109 raft_consensus.cc:2272] T a058316801f041a898362d1fdd5db080 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.350433 20109 tablet_replica.cc:333] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.350875 20109 raft_consensus.cc:2243] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.351234 20109 raft_consensus.cc:2272] T 691a85c9bd184f6788b51c19d2ad7273 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.352936 20109 tablet_replica.cc:333] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.353518 20109 raft_consensus.cc:2243] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.354393 20109 raft_consensus.cc:2272] T afd4e1cf18544f84a1a61cb7df061b74 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.355959 20109 tablet_replica.cc:333] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.356411 20109 raft_consensus.cc:2243] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 2 LEADER]: Raft consensus shutting down.
W20251215 08:58:20.356855 20874 consensus_peers.cc:597] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 -> Peer 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293): Couldn't send request to peer 6282ff6904b948e59e40f2db52b2be83. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:20.357178 20109 raft_consensus.cc:2272] T ccaea97f463c4731b8e73ee3d0028807 P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.358739 20109 tablet_replica.cc:333] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.359192 20109 raft_consensus.cc:2243] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.359563 20109 raft_consensus.cc:2272] T 1b8468317e1e402aba91c174231624e2 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.361091 20109 tablet_replica.cc:333] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.361563 20109 raft_consensus.cc:2243] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.361946 20109 raft_consensus.cc:2272] T a34845828f9a42118a523af87e308b08 P 6282ff6904b948e59e40f2db52b2be83 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.363628 20109 tablet_replica.cc:333] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.364064 20109 raft_consensus.cc:2243] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.364457 20109 raft_consensus.cc:2272] T 576271add9c246529e6deb72528494f1 P 6282ff6904b948e59e40f2db52b2be83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.366110 20109 tablet_replica.cc:333] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.366534 20109 raft_consensus.cc:2243] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.366930 20109 raft_consensus.cc:2272] T dc67dabaa62248d28af779466ba95976 P 6282ff6904b948e59e40f2db52b2be83 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.368471 20109 tablet_replica.cc:333] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83: stopping tablet replica
I20251215 08:58:20.368881 20109 raft_consensus.cc:2243] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.369272 20109 raft_consensus.cc:2272] T fd7013ed60464deca91ae876affc032d P 6282ff6904b948e59e40f2db52b2be83 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251215 08:58:20.381455 20872 consensus_peers.cc:597] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 -> Peer 2b0986965d0d49bf87a9aa87d2f06643 (127.19.163.65:43261): Couldn't send request to peer 2b0986965d0d49bf87a9aa87d2f06643. Status: Network error: Client connection negotiation failed: client connection to 127.19.163.65:43261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251215 08:58:20.406966 20874 consensus_peers.cc:597] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 -> Peer 6282ff6904b948e59e40f2db52b2be83 (127.19.163.66:45293): Couldn't send request to peer 6282ff6904b948e59e40f2db52b2be83. Status: Network error: Client connection negotiation failed: client connection to 127.19.163.66:45293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:20.417577 20109 tablet_server.cc:195] TabletServer@127.19.163.66:0 shutdown complete.
I20251215 08:58:20.436936 20109 tablet_server.cc:178] TabletServer@127.19.163.67:0 shutting down...
I20251215 08:58:20.463522 20109 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 08:58:20.464169 20109 tablet_replica.cc:333] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.464797 20109 raft_consensus.cc:2243] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.465312 20109 raft_consensus.cc:2272] T 6807ea1cd51f4491aaa320e8382f23e1 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.466988 20109 tablet_replica.cc:333] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.467440 20109 raft_consensus.cc:2243] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.467820 20109 raft_consensus.cc:2272] T fd7013ed60464deca91ae876affc032d P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.469494 20109 tablet_replica.cc:333] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.469911 20109 raft_consensus.cc:2243] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 4 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.470595 20109 raft_consensus.cc:2272] T 1b8468317e1e402aba91c174231624e2 P 48d193db12ff4e5b973328011a578553 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.472096 20109 tablet_replica.cc:333] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.472541 20109 raft_consensus.cc:2243] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 3 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.473270 20109 raft_consensus.cc:2272] T a34845828f9a42118a523af87e308b08 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.474702 20109 tablet_replica.cc:333] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.475106 20109 raft_consensus.cc:2243] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.475458 20109 raft_consensus.cc:2272] T a058316801f041a898362d1fdd5db080 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.476974 20109 tablet_replica.cc:333] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.477459 20109 raft_consensus.cc:2243] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 1 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.478103 20109 raft_consensus.cc:2272] T 576271add9c246529e6deb72528494f1 P 48d193db12ff4e5b973328011a578553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.479709 20109 tablet_replica.cc:333] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.480110 20109 raft_consensus.cc:2243] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.480443 20109 raft_consensus.cc:2272] T dc67dabaa62248d28af779466ba95976 P 48d193db12ff4e5b973328011a578553 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.481999 20109 tablet_replica.cc:333] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.482409 20109 raft_consensus.cc:2243] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.482787 20109 raft_consensus.cc:2272] T ccaea97f463c4731b8e73ee3d0028807 P 48d193db12ff4e5b973328011a578553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.484354 20109 tablet_replica.cc:333] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.484728 20109 raft_consensus.cc:2243] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.485072 20109 raft_consensus.cc:2272] T afd4e1cf18544f84a1a61cb7df061b74 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.486541 20109 tablet_replica.cc:333] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553: stopping tablet replica
I20251215 08:58:20.486900 20109 raft_consensus.cc:2243] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:20.487236 20109 raft_consensus.cc:2272] T 691a85c9bd184f6788b51c19d2ad7273 P 48d193db12ff4e5b973328011a578553 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.514729 20109 tablet_server.cc:195] TabletServer@127.19.163.67:0 shutdown complete.
I20251215 08:58:20.535198 20109 master.cc:561] Master@127.19.163.126:38021 shutting down...
I20251215 08:58:20.562624 20109 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 1 LEADER]: Raft consensus shutting down.
I20251215 08:58:20.563186 20109 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:20.563491 20109 tablet_replica.cc:333] T 00000000000000000000000000000000 P 742723155f494a569008014b9e0c3e75: stopping tablet replica
I20251215 08:58:20.583441 20109 master.cc:583] Master@127.19.163.126:38021 shutdown complete.
2b0986965d0d49bf87a9aa87d2f06643 5
48d193db12ff4e5b973328011a578553 1
6282ff6904b948e59e40f2db52b2be83 4
2b0986965d0d49bf87a9aa87d2f06643 3
48d193db12ff4e5b973328011a578553 3
6282ff6904b948e59e40f2db52b2be83 4
2b0986965d0d49bf87a9aa87d2f06643 8
48d193db12ff4e5b973328011a578553 1
6282ff6904b948e59e40f2db52b2be83 1
2b0986965d0d49bf87a9aa87d2f06643 4
48d193db12ff4e5b973328011a578553 3
6282ff6904b948e59e40f2db52b2be83 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28715 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20251215 08:58:20.622486 20109 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.163.126:33983
I20251215 08:58:20.623533 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:58:20.628913 21139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:58:20.629160 21140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:58:20.629719 21142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:20.630532 20109 server_base.cc:1047] running on GCE node
I20251215 08:58:20.631428 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:58:20.631637 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:58:20.631773 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789100631756 us; error 0 us; skew 500 ppm
I20251215 08:58:20.632226 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:58:20.634621 20109 webserver.cc:492] Webserver started at http://127.19.163.126:35751/ using document root <none> and password file <none>
I20251215 08:58:20.635062 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:58:20.635236 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:58:20.635474 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:58:20.636559 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/master-0-root/instance:
uuid: "1b8aeedd216e479da2a4e44a3dd8b23f"
format_stamp: "Formatted at 2025-12-15 08:58:20 on dist-test-slave-s29j"
I20251215 08:58:20.640758 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20251215 08:58:20.643905 21147 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:20.644632 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251215 08:58:20.644868 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/master-0-root
uuid: "1b8aeedd216e479da2a4e44a3dd8b23f"
format_stamp: "Formatted at 2025-12-15 08:58:20 on dist-test-slave-s29j"
I20251215 08:58:20.645141 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:58:20.663103 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:58:20.664228 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:58:20.701795 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.126:33983
I20251215 08:58:20.701889 21198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.126:33983 every 8 connection(s)
I20251215 08:58:20.705762 21199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:20.715766 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f: Bootstrap starting.
I20251215 08:58:20.720402 21199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:20.724627 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f: No bootstrap required, opened a new log
I20251215 08:58:20.726830 21199 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" member_type: VOTER }
I20251215 08:58:20.727257 21199 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:20.727506 21199 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8aeedd216e479da2a4e44a3dd8b23f, State: Initialized, Role: FOLLOWER
I20251215 08:58:20.728029 21199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" member_type: VOTER }
I20251215 08:58:20.728474 21199 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 08:58:20.728700 21199 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 08:58:20.728956 21199 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:20.733155 21199 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" member_type: VOTER }
I20251215 08:58:20.733659 21199 leader_election.cc:304] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8aeedd216e479da2a4e44a3dd8b23f; no voters:
I20251215 08:58:20.734817 21199 leader_election.cc:290] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 08:58:20.735177 21202 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:20.736513 21202 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [term 1 LEADER]: Becoming Leader. State: Replica: 1b8aeedd216e479da2a4e44a3dd8b23f, State: Running, Role: LEADER
I20251215 08:58:20.737182 21202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" member_type: VOTER }
I20251215 08:58:20.737906 21199 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [sys.catalog]: configured and running, proceeding with master startup.
I20251215 08:58:20.741891 21204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8aeedd216e479da2a4e44a3dd8b23f. Latest consensus state: current_term: 1 leader_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" member_type: VOTER } }
I20251215 08:58:20.742197 21203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8aeedd216e479da2a4e44a3dd8b23f" member_type: VOTER } }
I20251215 08:58:20.742607 21204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [sys.catalog]: This master's current role is: LEADER
I20251215 08:58:20.742908 21203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f [sys.catalog]: This master's current role is: LEADER
I20251215 08:58:20.746920 21209 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 08:58:20.755314 21209 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 08:58:20.757754 20109 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 08:58:20.764753 21209 catalog_manager.cc:1357] Generated new cluster ID: 3e07492e4e194daab7686ee91cc397ca
I20251215 08:58:20.765065 21209 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 08:58:20.789340 21209 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 08:58:20.791402 21209 catalog_manager.cc:1514] Loading token signing keys...
I20251215 08:58:20.807185 21209 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f: Generated new TSK 0
I20251215 08:58:20.807792 21209 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 08:58:20.823997 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:58:20.830018 21220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:58:20.830401 21221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:20.832831 20109 server_base.cc:1047] running on GCE node
W20251215 08:58:20.833062 21223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:20.833971 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:58:20.834157 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:58:20.834318 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789100834298 us; error 0 us; skew 500 ppm
I20251215 08:58:20.834856 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:58:20.837136 20109 webserver.cc:492] Webserver started at http://127.19.163.65:38591/ using document root <none> and password file <none>
I20251215 08:58:20.837604 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:58:20.837765 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:58:20.838014 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:58:20.839147 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/instance:
uuid: "beefe15185ac42ef9a20251007f39a56"
format_stamp: "Formatted at 2025-12-15 08:58:20 on dist-test-slave-s29j"
I20251215 08:58:20.843509 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20251215 08:58:20.846802 21228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:20.847509 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 08:58:20.847779 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root
uuid: "beefe15185ac42ef9a20251007f39a56"
format_stamp: "Formatted at 2025-12-15 08:58:20 on dist-test-slave-s29j"
I20251215 08:58:20.848034 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:58:20.868063 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:58:20.869377 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:58:20.870846 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:58:20.873178 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:58:20.873363 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:20.873625 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:58:20.873782 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:20.913375 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.65:46193
I20251215 08:58:20.913471 21290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.65:46193 every 8 connection(s)
I20251215 08:58:20.918116 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:58:20.925473 21295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:58:20.927251 21296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:58:20.930405 21298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:20.930770 20109 server_base.cc:1047] running on GCE node
I20251215 08:58:20.932082 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:58:20.932346 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:58:20.932389 21291 heartbeater.cc:344] Connected to a master server at 127.19.163.126:33983
I20251215 08:58:20.932636 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789100932618 us; error 0 us; skew 500 ppm
I20251215 08:58:20.932844 21291 heartbeater.cc:461] Registering TS with master...
I20251215 08:58:20.933434 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:58:20.933773 21291 heartbeater.cc:507] Master 127.19.163.126:33983 requested a full tablet report, sending...
I20251215 08:58:20.936133 20109 webserver.cc:492] Webserver started at http://127.19.163.66:36951/ using document root <none> and password file <none>
I20251215 08:58:20.936235 21164 ts_manager.cc:194] Registered new tserver with Master: beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:20.936663 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:58:20.936890 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:58:20.937166 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:58:20.938105 21164 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36406
I20251215 08:58:20.938278 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/instance:
uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
format_stamp: "Formatted at 2025-12-15 08:58:20 on dist-test-slave-s29j"
I20251215 08:58:20.943563 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251215 08:58:20.946985 21303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:20.947746 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 08:58:20.948014 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root
uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
format_stamp: "Formatted at 2025-12-15 08:58:20 on dist-test-slave-s29j"
I20251215 08:58:20.948266 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:58:20.967969 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:58:20.969208 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:58:20.970623 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:58:20.972760 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:58:20.972947 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:20.973214 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:58:20.973373 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:21.012519 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.66:36421
I20251215 08:58:21.012627 21365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.66:36421 every 8 connection(s)
I20251215 08:58:21.017160 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:58:21.023288 21369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:58:21.025761 21370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:21.032627 21366 heartbeater.cc:344] Connected to a master server at 127.19.163.126:33983
I20251215 08:58:21.033149 21366 heartbeater.cc:461] Registering TS with master...
I20251215 08:58:21.033516 20109 server_base.cc:1047] running on GCE node
W20251215 08:58:21.033715 21372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:21.034217 21366 heartbeater.cc:507] Master 127.19.163.126:33983 requested a full tablet report, sending...
I20251215 08:58:21.034706 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:58:21.034925 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:58:21.035148 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789101035129 us; error 0 us; skew 500 ppm
I20251215 08:58:21.035777 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:58:21.036193 21164 ts_manager.cc:194] Registered new tserver with Master: 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:21.037686 21164 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36408
I20251215 08:58:21.038317 20109 webserver.cc:492] Webserver started at http://127.19.163.67:40063/ using document root <none> and password file <none>
I20251215 08:58:21.038899 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:58:21.039074 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:58:21.039309 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:58:21.040390 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/instance:
uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
format_stamp: "Formatted at 2025-12-15 08:58:21 on dist-test-slave-s29j"
I20251215 08:58:21.044801 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251215 08:58:21.047993 21377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:21.048722 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251215 08:58:21.049037 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root
uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
format_stamp: "Formatted at 2025-12-15 08:58:21 on dist-test-slave-s29j"
I20251215 08:58:21.049302 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:58:21.061089 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:58:21.062319 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:58:21.063679 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:58:21.065887 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:58:21.066068 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:21.066323 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:58:21.066463 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:21.105248 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.67:37447
I20251215 08:58:21.105358 21439 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.67:37447 every 8 connection(s)
I20251215 08:58:21.118451 21440 heartbeater.cc:344] Connected to a master server at 127.19.163.126:33983
I20251215 08:58:21.118800 21440 heartbeater.cc:461] Registering TS with master...
I20251215 08:58:21.119546 21440 heartbeater.cc:507] Master 127.19.163.126:33983 requested a full tablet report, sending...
I20251215 08:58:21.121456 21164 ts_manager.cc:194] Registered new tserver with Master: 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:21.121623 20109 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013407772s
I20251215 08:58:21.123219 21164 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36424
I20251215 08:58:21.124667 20109 test_util.cc:276] Using random seed: -803279837
I20251215 08:58:21.146612 21164 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36434:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 08:58:21.153095 21164 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 08:58:21.354775 21405 tablet_service.cc:1505] Processing CreateTablet for tablet a76bbffe260349a49c7cad7a3de2a9da (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251215 08:58:21.356235 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76bbffe260349a49c7cad7a3de2a9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.357657 21402 tablet_service.cc:1505] Processing CreateTablet for tablet caf0c0cd96174b4889e367b1f98d617e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251215 08:58:21.357880 21331 tablet_service.cc:1505] Processing CreateTablet for tablet a76bbffe260349a49c7cad7a3de2a9da (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251215 08:58:21.358939 21256 tablet_service.cc:1505] Processing CreateTablet for tablet a76bbffe260349a49c7cad7a3de2a9da (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251215 08:58:21.360263 21256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76bbffe260349a49c7cad7a3de2a9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.359937 21401 tablet_service.cc:1505] Processing CreateTablet for tablet dd5a10a353d64f5c826c27e4a01f6132 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251215 08:58:21.360680 21330 tablet_service.cc:1505] Processing CreateTablet for tablet 109b80fba7b544e9946db3216c2c7581 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251215 08:58:21.362119 21403 tablet_service.cc:1505] Processing CreateTablet for tablet 0c3a05444c334cad8e7562a15e4d68ba (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251215 08:58:21.362946 21329 tablet_service.cc:1505] Processing CreateTablet for tablet 0c3a05444c334cad8e7562a15e4d68ba (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251215 08:58:21.364224 21404 tablet_service.cc:1505] Processing CreateTablet for tablet 109b80fba7b544e9946db3216c2c7581 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251215 08:58:21.359345 21331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76bbffe260349a49c7cad7a3de2a9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.366426 21328 tablet_service.cc:1505] Processing CreateTablet for tablet caf0c0cd96174b4889e367b1f98d617e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251215 08:58:21.369181 21404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109b80fba7b544e9946db3216c2c7581. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.368940 21326 tablet_service.cc:1505] Processing CreateTablet for tablet e80fc34b30d346baa0dfec4df957cd9f (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251215 08:58:21.370321 21326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80fc34b30d346baa0dfec4df957cd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.373623 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c3a05444c334cad8e7562a15e4d68ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.373914 21327 tablet_service.cc:1505] Processing CreateTablet for tablet dd5a10a353d64f5c826c27e4a01f6132 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251215 08:58:21.374886 21401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5a10a353d64f5c826c27e4a01f6132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.375245 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5a10a353d64f5c826c27e4a01f6132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.376358 21402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf0c0cd96174b4889e367b1f98d617e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.376470 21328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf0c0cd96174b4889e367b1f98d617e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.377727 21255 tablet_service.cc:1505] Processing CreateTablet for tablet 109b80fba7b544e9946db3216c2c7581 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251215 08:58:21.379184 21255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109b80fba7b544e9946db3216c2c7581. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.379817 21325 tablet_service.cc:1505] Processing CreateTablet for tablet 3d471c9f4eb74ea3a20cec86631e4ab0 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251215 08:58:21.380062 21324 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9ae02f9a5345069e7afbf6d52e9299 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251215 08:58:21.377701 21330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109b80fba7b544e9946db3216c2c7581. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.381704 21325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d471c9f4eb74ea3a20cec86631e4ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.382455 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c3a05444c334cad8e7562a15e4d68ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.382109 21323 tablet_service.cc:1505] Processing CreateTablet for tablet 080eec653ea04179a0dceafb6e3ffe47 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251215 08:58:21.384037 21323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080eec653ea04179a0dceafb6e3ffe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.385818 21324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9ae02f9a5345069e7afbf6d52e9299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.394807 21254 tablet_service.cc:1505] Processing CreateTablet for tablet 0c3a05444c334cad8e7562a15e4d68ba (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251215 08:58:21.396350 21254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c3a05444c334cad8e7562a15e4d68ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.403169 21322 tablet_service.cc:1505] Processing CreateTablet for tablet 641cd3a624114cf98d427a39378b6da6 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251215 08:58:21.404623 21322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641cd3a624114cf98d427a39378b6da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.408407 21253 tablet_service.cc:1505] Processing CreateTablet for tablet caf0c0cd96174b4889e367b1f98d617e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251215 08:58:21.409849 21253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf0c0cd96174b4889e367b1f98d617e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.422832 21252 tablet_service.cc:1505] Processing CreateTablet for tablet dd5a10a353d64f5c826c27e4a01f6132 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251215 08:58:21.424319 21252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5a10a353d64f5c826c27e4a01f6132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.486414 21251 tablet_service.cc:1505] Processing CreateTablet for tablet e80fc34b30d346baa0dfec4df957cd9f (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251215 08:58:21.487879 21251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80fc34b30d346baa0dfec4df957cd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.495018 21250 tablet_service.cc:1505] Processing CreateTablet for tablet 3d471c9f4eb74ea3a20cec86631e4ab0 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251215 08:58:21.496791 21250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d471c9f4eb74ea3a20cec86631e4ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.506237 21249 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9ae02f9a5345069e7afbf6d52e9299 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251215 08:58:21.508060 21249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9ae02f9a5345069e7afbf6d52e9299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.518477 21400 tablet_service.cc:1505] Processing CreateTablet for tablet e80fc34b30d346baa0dfec4df957cd9f (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251215 08:58:21.519951 21400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80fc34b30d346baa0dfec4df957cd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.520921 21322 tablet_service.cc:1505] Processing CreateTablet for tablet d8b3d79d7a014e568e385110044efb63 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251215 08:58:21.521133 21324 tablet_service.cc:1505] Processing CreateTablet for tablet 01ad5646d4d04a6a996ce407925b8650 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251215 08:58:21.522259 21322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b3d79d7a014e568e385110044efb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.525936 21324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ad5646d4d04a6a996ce407925b8650. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.535411 21248 tablet_service.cc:1505] Processing CreateTablet for tablet 080eec653ea04179a0dceafb6e3ffe47 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251215 08:58:21.537797 21247 tablet_service.cc:1505] Processing CreateTablet for tablet 641cd3a624114cf98d427a39378b6da6 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251215 08:58:21.548734 21398 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9ae02f9a5345069e7afbf6d52e9299 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251215 08:58:21.550146 21398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9ae02f9a5345069e7afbf6d52e9299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.548733 21399 tablet_service.cc:1505] Processing CreateTablet for tablet 3d471c9f4eb74ea3a20cec86631e4ab0 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251215 08:58:21.561499 21399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d471c9f4eb74ea3a20cec86631e4ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.562235 21397 tablet_service.cc:1505] Processing CreateTablet for tablet 080eec653ea04179a0dceafb6e3ffe47 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251215 08:58:21.563627 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080eec653ea04179a0dceafb6e3ffe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.567158 21396 tablet_service.cc:1505] Processing CreateTablet for tablet 641cd3a624114cf98d427a39378b6da6 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251215 08:58:21.568548 21396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641cd3a624114cf98d427a39378b6da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.585080 21248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080eec653ea04179a0dceafb6e3ffe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.588955 21325 tablet_service.cc:1505] Processing CreateTablet for tablet 9b36e018bdd140648e80e809a47269c8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251215 08:58:21.589416 21326 tablet_service.cc:1505] Processing CreateTablet for tablet 9d1675f3765f488e9d58a2ba2993457e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251215 08:58:21.590328 21325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b36e018bdd140648e80e809a47269c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.595223 21328 tablet_service.cc:1505] Processing CreateTablet for tablet 0922c97d6dce49f0a3b98d200f4f651e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251215 08:58:21.602386 21405 tablet_service.cc:1505] Processing CreateTablet for tablet 01ad5646d4d04a6a996ce407925b8650 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251215 08:58:21.603755 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ad5646d4d04a6a996ce407925b8650. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.596489 21328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0922c97d6dce49f0a3b98d200f4f651e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.613935 21401 tablet_service.cc:1505] Processing CreateTablet for tablet d8b3d79d7a014e568e385110044efb63 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251215 08:58:21.614936 21401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b3d79d7a014e568e385110044efb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.617826 21456 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:21.622011 21404 tablet_service.cc:1505] Processing CreateTablet for tablet 9b36e018bdd140648e80e809a47269c8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251215 08:58:21.623835 21404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b36e018bdd140648e80e809a47269c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.627972 21254 tablet_service.cc:1505] Processing CreateTablet for tablet d8b3d79d7a014e568e385110044efb63 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251215 08:58:21.642835 21252 tablet_service.cc:1505] Processing CreateTablet for tablet 01ad5646d4d04a6a996ce407925b8650 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251215 08:58:21.656641 21326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1675f3765f488e9d58a2ba2993457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.663807 21457 tablet_bootstrap.cc:492] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:21.665591 21396 tablet_service.cc:1505] Processing CreateTablet for tablet 9d1675f3765f488e9d58a2ba2993457e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251215 08:58:21.665994 21400 tablet_service.cc:1505] Processing CreateTablet for tablet 0922c97d6dce49f0a3b98d200f4f651e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251215 08:58:21.666877 21396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1675f3765f488e9d58a2ba2993457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.671988 21402 tablet_service.cc:1505] Processing CreateTablet for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251215 08:58:21.673382 21402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.675568 21398 tablet_service.cc:1505] Processing CreateTablet for tablet 5116dbb730274ac9982610c7f5f8f5cd (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251215 08:58:21.676795 21398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5116dbb730274ac9982610c7f5f8f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.683828 21457 tablet_bootstrap.cc:654] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:21.685901 21456 tablet_bootstrap.cc:654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:21.688556 21396 tablet_service.cc:1505] Processing CreateTablet for tablet ac8397b87ffc453b8f468eee77a8e410 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251215 08:58:21.689813 21396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac8397b87ffc453b8f468eee77a8e410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.722468 21249 tablet_service.cc:1505] Processing CreateTablet for tablet 9d1675f3765f488e9d58a2ba2993457e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251215 08:58:21.729676 21247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641cd3a624114cf98d427a39378b6da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.662122 21256 tablet_service.cc:1505] Processing CreateTablet for tablet 9b36e018bdd140648e80e809a47269c8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251215 08:58:21.739020 21256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b36e018bdd140648e80e809a47269c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.742317 21252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ad5646d4d04a6a996ce407925b8650. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.747625 21324 tablet_service.cc:1505] Processing CreateTablet for tablet 5116dbb730274ac9982610c7f5f8f5cd (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251215 08:58:21.749276 21324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5116dbb730274ac9982610c7f5f8f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.749550 21254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b3d79d7a014e568e385110044efb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.759519 21325 tablet_service.cc:1505] Processing CreateTablet for tablet 88823d4c370d4c6a96a6ba54418638eb (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251215 08:58:21.760977 21325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88823d4c370d4c6a96a6ba54418638eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251215 08:58:21.756806 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:21.769449 21249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1675f3765f488e9d58a2ba2993457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.782393 21328 tablet_service.cc:1505] Processing CreateTablet for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251215 08:58:21.783833 21328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.788187 21322 tablet_service.cc:1505] Processing CreateTablet for tablet c61929339a494e3fb5f327784d94d817 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251215 08:58:21.789697 21322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61929339a494e3fb5f327784d94d817. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.794270 21329 tablet_service.cc:1505] Processing CreateTablet for tablet f14cfff8c5a44ad78299ada0ca53baa9 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251215 08:58:21.795832 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14cfff8c5a44ad78299ada0ca53baa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.802428 21330 tablet_service.cc:1505] Processing CreateTablet for tablet ba393161b7454821b7584a44b64bf6b2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251215 08:58:21.803167 21327 tablet_service.cc:1505] Processing CreateTablet for tablet 676d761e0efc443cad981187af43b8d9 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251215 08:58:21.803880 21330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba393161b7454821b7584a44b64bf6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.804497 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676d761e0efc443cad981187af43b8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.806048 21331 tablet_service.cc:1505] Processing CreateTablet for tablet ac8397b87ffc453b8f468eee77a8e410 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251215 08:58:21.807339 21331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac8397b87ffc453b8f468eee77a8e410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.822474 21400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0922c97d6dce49f0a3b98d200f4f651e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.823562 21401 tablet_service.cc:1505] Processing CreateTablet for tablet 88823d4c370d4c6a96a6ba54418638eb (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251215 08:58:21.824995 21401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88823d4c370d4c6a96a6ba54418638eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.831633 21458 tablet_bootstrap.cc:492] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:21.840797 21399 tablet_service.cc:1505] Processing CreateTablet for tablet c61929339a494e3fb5f327784d94d817 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251215 08:58:21.830639 21247 tablet_service.cc:1505] Processing CreateTablet for tablet 5116dbb730274ac9982610c7f5f8f5cd (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251215 08:58:21.842254 21399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61929339a494e3fb5f327784d94d817. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.842322 21247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5116dbb730274ac9982610c7f5f8f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.828660 21250 tablet_service.cc:1505] Processing CreateTablet for tablet 0922c97d6dce49f0a3b98d200f4f651e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251215 08:58:21.827739 21398 tablet_service.cc:1505] Processing CreateTablet for tablet f14cfff8c5a44ad78299ada0ca53baa9 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251215 08:58:21.826853 21256 tablet_service.cc:1505] Processing CreateTablet for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251215 08:58:21.830751 21251 tablet_service.cc:1505] Processing CreateTablet for tablet ac8397b87ffc453b8f468eee77a8e410 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251215 08:58:21.826467 21397 tablet_service.cc:1505] Processing CreateTablet for tablet ba393161b7454821b7584a44b64bf6b2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251215 08:58:21.856177 21405 tablet_service.cc:1505] Processing CreateTablet for tablet 676d761e0efc443cad981187af43b8d9 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251215 08:58:21.869745 21323 tablet_service.cc:1505] Processing CreateTablet for tablet 6053bc9278744c6b93f60f788aa7b06e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251215 08:58:21.870163 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba393161b7454821b7584a44b64bf6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.870508 21403 tablet_service.cc:1505] Processing CreateTablet for tablet 067a0581bd0244539c1eca2c554e4619 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251215 08:58:21.870991 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676d761e0efc443cad981187af43b8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.871201 21323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6053bc9278744c6b93f60f788aa7b06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.871887 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067a0581bd0244539c1eca2c554e4619. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.873131 21253 tablet_service.cc:1505] Processing CreateTablet for tablet ba393161b7454821b7584a44b64bf6b2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251215 08:58:21.868863 21250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0922c97d6dce49f0a3b98d200f4f651e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.875175 21456 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:21.875836 21456 ts_tablet_manager.cc:1403] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.258s user 0.010s sys 0.000s
I20251215 08:58:21.884349 21398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14cfff8c5a44ad78299ada0ca53baa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.887537 21456 raft_consensus.cc:359] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:21.888677 21456 raft_consensus.cc:385] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:21.884843 21331 tablet_service.cc:1505] Processing CreateTablet for tablet 59576dccc86c4396bdbb3706a9750747 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251215 08:58:21.880251 21255 tablet_service.cc:1505] Processing CreateTablet for tablet 88823d4c370d4c6a96a6ba54418638eb (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251215 08:58:21.899132 21331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59576dccc86c4396bdbb3706a9750747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.920286 21251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac8397b87ffc453b8f468eee77a8e410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.931138 21405 tablet_service.cc:1505] Processing CreateTablet for tablet 6053bc9278744c6b93f60f788aa7b06e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251215 08:58:21.932499 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6053bc9278744c6b93f60f788aa7b06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.941197 21291 heartbeater.cc:499] Master 127.19.163.126:33983 was elected leader, sending a full tablet report...
I20251215 08:58:21.942641 21329 tablet_service.cc:1505] Processing CreateTablet for tablet b7bfa9bddb4b48d996b8b347c898a3b7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251215 08:58:21.953864 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7bfa9bddb4b48d996b8b347c898a3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.954209 21396 tablet_service.cc:1505] Processing CreateTablet for tablet 59576dccc86c4396bdbb3706a9750747 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251215 08:58:21.955461 21396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59576dccc86c4396bdbb3706a9750747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.945551 21458 tablet_bootstrap.cc:654] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:21.962345 21397 tablet_service.cc:1505] Processing CreateTablet for tablet bfcf2c8cd22a488aa741f1cede6e86ef (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251215 08:58:21.962977 21405 tablet_service.cc:1505] Processing CreateTablet for tablet b7bfa9bddb4b48d996b8b347c898a3b7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251215 08:58:21.963814 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfcf2c8cd22a488aa741f1cede6e86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.965728 21253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba393161b7454821b7584a44b64bf6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.969318 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7bfa9bddb4b48d996b8b347c898a3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.969532 21255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88823d4c370d4c6a96a6ba54418638eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.970680 21254 tablet_service.cc:1505] Processing CreateTablet for tablet f14cfff8c5a44ad78299ada0ca53baa9 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251215 08:58:21.971894 21254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14cfff8c5a44ad78299ada0ca53baa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.981369 21457 tablet_bootstrap.cc:492] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:21.982033 21457 ts_tablet_manager.cc:1403] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.320s user 0.011s sys 0.013s
I20251215 08:58:21.985596 21256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.993655 21327 tablet_service.cc:1505] Processing CreateTablet for tablet 067a0581bd0244539c1eca2c554e4619 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251215 08:58:21.995060 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067a0581bd0244539c1eca2c554e4619. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.888986 21456 raft_consensus.cc:740] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.001832 21456 consensus_queue.cc:260] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.006582 21254 tablet_service.cc:1505] Processing CreateTablet for tablet c61929339a494e3fb5f327784d94d817 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251215 08:58:22.016355 21254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61929339a494e3fb5f327784d94d817. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.019228 21402 tablet_service.cc:1505] Processing CreateTablet for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251215 08:58:22.020260 21249 tablet_service.cc:1505] Processing CreateTablet for tablet 067a0581bd0244539c1eca2c554e4619 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251215 08:58:22.020818 21402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.021147 21326 tablet_service.cc:1505] Processing CreateTablet for tablet f757e923d11a495091c94621efa6e4c1 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251215 08:58:22.019351 21397 tablet_service.cc:1505] Processing CreateTablet for tablet 5d93e1c6de78493c926c5be7096cfb31 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251215 08:58:22.017639 21405 tablet_service.cc:1505] Processing CreateTablet for tablet f757e923d11a495091c94621efa6e4c1 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251215 08:58:22.027136 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d93e1c6de78493c926c5be7096cfb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.029402 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f757e923d11a495091c94621efa6e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.024094 21329 tablet_service.cc:1505] Processing CreateTablet for tablet 60989b82e8354098aac2066e31697953 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251215 08:58:22.029963 21330 tablet_service.cc:1505] Processing CreateTablet for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251215 08:58:22.031802 21325 tablet_service.cc:1505] Processing CreateTablet for tablet 7edcdb5749ae4e59ab17648fe97390e2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251215 08:58:22.040405 21366 heartbeater.cc:499] Master 127.19.163.126:33983 was elected leader, sending a full tablet report...
I20251215 08:58:22.014227 21248 tablet_service.cc:1505] Processing CreateTablet for tablet 676d761e0efc443cad981187af43b8d9 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251215 08:58:22.041931 21404 tablet_service.cc:1505] Processing CreateTablet for tablet 60989b82e8354098aac2066e31697953 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251215 08:58:22.062985 21404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60989b82e8354098aac2066e31697953. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:21.956547 21323 tablet_service.cc:1505] Processing CreateTablet for tablet bfcf2c8cd22a488aa741f1cede6e86ef (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251215 08:58:22.045787 21403 tablet_service.cc:1505] Processing CreateTablet for tablet f800ccb361734ac18102a3502ee639d8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251215 08:58:22.052316 21457 raft_consensus.cc:359] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:22.070597 21457 raft_consensus.cc:385] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:22.070976 21457 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.071802 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f800ccb361734ac18102a3502ee639d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.061158 21398 tablet_service.cc:1505] Processing CreateTablet for tablet 64b5a2ef4843492085aca418ad649914 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251215 08:58:22.073765 21398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b5a2ef4843492085aca418ad649914. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.074846 21400 tablet_service.cc:1505] Processing CreateTablet for tablet 458cec829de44303951b997be68accde (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251215 08:58:22.074988 21405 tablet_service.cc:1505] Processing CreateTablet for tablet 7edcdb5749ae4e59ab17648fe97390e2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251215 08:58:22.076182 21400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 458cec829de44303951b997be68accde. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.071709 21457 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:22.082005 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edcdb5749ae4e59ab17648fe97390e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.082710 21324 tablet_service.cc:1505] Processing CreateTablet for tablet 5d93e1c6de78493c926c5be7096cfb31 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251215 08:58:22.084767 21252 tablet_service.cc:1505] Processing CreateTablet for tablet 6053bc9278744c6b93f60f788aa7b06e (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251215 08:58:22.086697 21322 tablet_service.cc:1505] Processing CreateTablet for tablet f800ccb361734ac18102a3502ee639d8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251215 08:58:22.094038 21401 tablet_service.cc:1505] Processing CreateTablet for tablet 79790fd760ff4ae1b3879326decc531a (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251215 08:58:22.095459 21401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79790fd760ff4ae1b3879326decc531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.022415 21326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f757e923d11a495091c94621efa6e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.105978 21456 ts_tablet_manager.cc:1434] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.230s user 0.000s sys 0.016s
I20251215 08:58:22.109536 21250 tablet_service.cc:1505] Processing CreateTablet for tablet 59576dccc86c4396bdbb3706a9750747 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251215 08:58:22.110340 21328 tablet_service.cc:1505] Processing CreateTablet for tablet 79790fd760ff4ae1b3879326decc531a (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251215 08:58:22.112202 21249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067a0581bd0244539c1eca2c554e4619. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.112883 21248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676d761e0efc443cad981187af43b8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.114727 21405 tablet_service.cc:1505] Processing CreateTablet for tablet 8d7af00ee6834714a059e045b3c13322 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251215 08:58:22.115895 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d7af00ee6834714a059e045b3c13322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.119043 21404 tablet_service.cc:1505] Processing CreateTablet for tablet b3680af750ef47df88bf5be33eb18fc7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251215 08:58:22.119580 21250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59576dccc86c4396bdbb3706a9750747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.125373 21323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfcf2c8cd22a488aa741f1cede6e86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.125928 21440 heartbeater.cc:499] Master 127.19.163.126:33983 was elected leader, sending a full tablet report...
I20251215 08:58:22.134400 21404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3680af750ef47df88bf5be33eb18fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.137214 21324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d93e1c6de78493c926c5be7096cfb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.141341 21330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.143646 21252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6053bc9278744c6b93f60f788aa7b06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.145861 21456 tablet_bootstrap.cc:492] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:22.148193 21255 tablet_service.cc:1505] Processing CreateTablet for tablet bfcf2c8cd22a488aa741f1cede6e86ef (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251215 08:58:22.149852 21255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfcf2c8cd22a488aa741f1cede6e86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.152329 21399 tablet_service.cc:1505] Processing CreateTablet for tablet 0351e3362cb742aa87ffc4bf31ceea2c (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251215 08:58:22.142946 21253 tablet_service.cc:1505] Processing CreateTablet for tablet b7bfa9bddb4b48d996b8b347c898a3b7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251215 08:58:22.209901 21253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7bfa9bddb4b48d996b8b347c898a3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.154979 21247 tablet_service.cc:1505] Processing CreateTablet for tablet f757e923d11a495091c94621efa6e4c1 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251215 08:58:22.217873 21247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f757e923d11a495091c94621efa6e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.159437 21457 ts_tablet_manager.cc:1434] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.177s user 0.011s sys 0.014s
I20251215 08:58:22.221390 21399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0351e3362cb742aa87ffc4bf31ceea2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.160326 21251 tablet_service.cc:1505] Processing CreateTablet for tablet 60989b82e8354098aac2066e31697953 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251215 08:58:22.171824 21328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79790fd760ff4ae1b3879326decc531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.200599 21331 tablet_service.cc:1505] Processing CreateTablet for tablet 458cec829de44303951b997be68accde (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251215 08:58:22.223932 21331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 458cec829de44303951b997be68accde. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.179834 21458 tablet_bootstrap.cc:492] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:22.224673 21458 ts_tablet_manager.cc:1403] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.394s user 0.029s sys 0.050s
I20251215 08:58:22.225371 21325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edcdb5749ae4e59ab17648fe97390e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.226372 21322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f800ccb361734ac18102a3502ee639d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.197302 21462 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:22.229612 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60989b82e8354098aac2066e31697953. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.229249 21462 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.198290 21327 tablet_service.cc:1505] Processing CreateTablet for tablet 64b5a2ef4843492085aca418ad649914 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251215 08:58:22.233263 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b5a2ef4843492085aca418ad649914. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.199988 21254 tablet_service.cc:1505] Processing CreateTablet for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251215 08:58:22.202739 21397 tablet_service.cc:1505] Processing CreateTablet for tablet 07b6839b646d44fbac5ae9e9e2a65daa (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251215 08:58:22.235890 21254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.236110 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b6839b646d44fbac5ae9e9e2a65daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.245538 21457 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:22.255823 21457 tablet_bootstrap.cc:654] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.259335 21403 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1c639a483f4827997eae6f3a6e375d (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251215 08:58:22.260720 21402 tablet_service.cc:1505] Processing CreateTablet for tablet 124c9efc80174ca585aaee7e0f8a12f0 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251215 08:58:22.265815 21251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60989b82e8354098aac2066e31697953. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.266901 21458 raft_consensus.cc:359] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.270252 21323 tablet_service.cc:1505] Processing CreateTablet for tablet 8d7af00ee6834714a059e045b3c13322 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251215 08:58:22.260737 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1c639a483f4827997eae6f3a6e375d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.276530 21248 tablet_service.cc:1505] Processing CreateTablet for tablet 5d93e1c6de78493c926c5be7096cfb31 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251215 08:58:22.288776 21462 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:22.285693 21458 raft_consensus.cc:385] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:22.310271 21456 tablet_bootstrap.cc:654] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.311897 21252 tablet_service.cc:1505] Processing CreateTablet for tablet 64b5a2ef4843492085aca418ad649914 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251215 08:58:22.312354 21398 tablet_service.cc:1505] Processing CreateTablet for tablet 2c10c3de775846868774c98cc0a75094 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251215 08:58:22.300758 21458 raft_consensus.cc:740] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.314563 21463 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:22.314626 21458 consensus_queue.cc:260] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.315119 21463 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
W20251215 08:58:22.315894 21367 tablet.cc:2378] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 08:58:22.327518 21252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b5a2ef4843492085aca418ad649914. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.333873 21323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d7af00ee6834714a059e045b3c13322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.334224 21402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 124c9efc80174ca585aaee7e0f8a12f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.364462 21457 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:22.365108 21457 ts_tablet_manager.cc:1403] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.120s user 0.021s sys 0.012s
I20251215 08:58:22.368026 21457 raft_consensus.cc:359] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.368736 21457 raft_consensus.cc:385] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:22.369038 21457 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.360755 21248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d93e1c6de78493c926c5be7096cfb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.351995 21463 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:22.373291 21398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c10c3de775846868774c98cc0a75094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.391443 21458 ts_tablet_manager.cc:1434] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.166s user 0.008s sys 0.010s
I20251215 08:58:22.392573 21458 tablet_bootstrap.cc:492] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:22.395769 21253 tablet_service.cc:1505] Processing CreateTablet for tablet 458cec829de44303951b997be68accde (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251215 08:58:22.397982 21255 tablet_service.cc:1505] Processing CreateTablet for tablet 7edcdb5749ae4e59ab17648fe97390e2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251215 08:58:22.401984 21401 tablet_service.cc:1505] Processing CreateTablet for tablet 9582bb1875204089bca2e38a3ff426b3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251215 08:58:22.403229 21401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9582bb1875204089bca2e38a3ff426b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.407387 21331 tablet_service.cc:1505] Processing CreateTablet for tablet 07b6839b646d44fbac5ae9e9e2a65daa (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251215 08:58:22.409521 21404 tablet_service.cc:1505] Processing CreateTablet for tablet 15d0087ada21421eacebc646b271e145 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251215 08:58:22.409777 21396 tablet_service.cc:1505] Processing CreateTablet for tablet 7da883c18eeb4679974472b9b341b5e8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251215 08:58:22.410704 21324 tablet_service.cc:1505] Processing CreateTablet for tablet b3680af750ef47df88bf5be33eb18fc7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251215 08:58:22.411445 21396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da883c18eeb4679974472b9b341b5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.385190 21250 tablet_service.cc:1505] Processing CreateTablet for tablet 79790fd760ff4ae1b3879326decc531a (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251215 08:58:22.418313 21326 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1c639a483f4827997eae6f3a6e375d (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251215 08:58:22.421753 21247 tablet_service.cc:1505] Processing CreateTablet for tablet 8d7af00ee6834714a059e045b3c13322 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251215 08:58:22.422505 21256 tablet_service.cc:1505] Processing CreateTablet for tablet f800ccb361734ac18102a3502ee639d8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251215 08:58:22.385558 21457 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.431469 21457 ts_tablet_manager.cc:1434] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.066s user 0.005s sys 0.001s
I20251215 08:58:22.432482 21457 tablet_bootstrap.cc:492] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:22.442188 21405 tablet_service.cc:1505] Processing CreateTablet for tablet 84bdfb60f8fa4b39bc8cbd437a0809f2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251215 08:58:22.409498 21331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b6839b646d44fbac5ae9e9e2a65daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.455211 21325 tablet_service.cc:1505] Processing CreateTablet for tablet 124c9efc80174ca585aaee7e0f8a12f0 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251215 08:58:22.428614 21322 tablet_service.cc:1505] Processing CreateTablet for tablet 0351e3362cb742aa87ffc4bf31ceea2c (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251215 08:58:22.457249 21253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 458cec829de44303951b997be68accde. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.464087 21330 tablet_service.cc:1505] Processing CreateTablet for tablet 2c10c3de775846868774c98cc0a75094 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251215 08:58:22.465135 21322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0351e3362cb742aa87ffc4bf31ceea2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.448143 21404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d0087ada21421eacebc646b271e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.478648 21456 tablet_bootstrap.cc:492] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:22.478792 21256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f800ccb361734ac18102a3502ee639d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.487298 21399 tablet_service.cc:1505] Processing CreateTablet for tablet d69415d618114174b070bbc19a876119 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251215 08:58:22.488410 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:22.496547 21307 leader_election.cc:343] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.501700 21324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3680af750ef47df88bf5be33eb18fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.505347 21250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79790fd760ff4ae1b3879326decc531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.505492 21456 ts_tablet_manager.cc:1403] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.360s user 0.017s sys 0.016s
I20251215 08:58:22.509338 21463 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:22.510031 21463 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.512673 21463 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:22.513358 21247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d7af00ee6834714a059e045b3c13322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.513839 21399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69415d618114174b070bbc19a876119. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.516136 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bdfb60f8fa4b39bc8cbd437a0809f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.516175 21400 tablet_service.cc:1505] Processing CreateTablet for tablet 1db12563817b44429a055b7dcdd066c3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251215 08:58:22.517727 21400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db12563817b44429a055b7dcdd066c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.520212 21255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edcdb5749ae4e59ab17648fe97390e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.521311 21397 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf7b0165c00477b9c1b94c6a89052a6 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251215 08:58:22.522567 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf7b0165c00477b9c1b94c6a89052a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.484613 21254 tablet_service.cc:1505] Processing CreateTablet for tablet b3680af750ef47df88bf5be33eb18fc7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251215 08:58:22.523658 21254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3680af750ef47df88bf5be33eb18fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.527240 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:22.528281 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:22.508562 21456 raft_consensus.cc:359] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:22.529482 21381 leader_election.cc:343] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:22.530182 21381 leader_election.cc:343] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.529484 21456 raft_consensus.cc:385] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:22.531100 21456 raft_consensus.cc:740] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.532862 21249 tablet_service.cc:1505] Processing CreateTablet for tablet 07b6839b646d44fbac5ae9e9e2a65daa (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251215 08:58:22.534102 21249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b6839b646d44fbac5ae9e9e2a65daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.541383 21456 consensus_queue.cc:260] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.521317 21403 tablet_service.cc:1505] Processing CreateTablet for tablet 4c50acc4d0854bd08c6afc283d478ca7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251215 08:58:22.547150 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:22.549764 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:22.550556 21462 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:58:22.554987 21379 leader_election.cc:343] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.554689 21462 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.558192 21462 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:22.557914 21328 tablet_service.cc:1505] Processing CreateTablet for tablet 1db12563817b44429a055b7dcdd066c3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251215 08:58:22.559178 21325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 124c9efc80174ca585aaee7e0f8a12f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.560236 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:22.561906 21307 leader_election.cc:343] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.562633 21379 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:22.563508 21323 tablet_service.cc:1505] Processing CreateTablet for tablet 15d0087ada21421eacebc646b271e145 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251215 08:58:22.564399 21463 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251215 08:58:22.563962 21379 leader_election.cc:343] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.565688 21379 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:22.566102 21457 tablet_bootstrap.cc:654] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.555022 21327 tablet_service.cc:1505] Processing CreateTablet for tablet 7da883c18eeb4679974472b9b341b5e8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251215 08:58:22.566812 21329 tablet_service.cc:1505] Processing CreateTablet for tablet 9582bb1875204089bca2e38a3ff426b3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251215 08:58:22.567695 21463 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:22.572264 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9582bb1875204089bca2e38a3ff426b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.585736 21251 tablet_service.cc:1505] Processing CreateTablet for tablet 15d0087ada21421eacebc646b271e145 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251215 08:58:22.587185 21251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d0087ada21421eacebc646b271e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.588567 21330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c10c3de775846868774c98cc0a75094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.594347 21456 ts_tablet_manager.cc:1434] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.088s user 0.001s sys 0.006s
I20251215 08:58:22.595508 21456 tablet_bootstrap.cc:492] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:22.597354 21328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db12563817b44429a055b7dcdd066c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.598662 21398 tablet_service.cc:1505] Processing CreateTablet for tablet 4d1d9a9be078441587cf5da752d59c33 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251215 08:58:22.608071 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:22.609730 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:22.611487 21307 leader_election.cc:343] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.611940 21307 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:22.578871 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c50acc4d0854bd08c6afc283d478ca7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251215 08:58:22.613240 21307 leader_election.cc:343] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.613673 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:22.614418 21462 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:22.614954 21462 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:22.604429 21326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1c639a483f4827997eae6f3a6e375d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.620736 21396 tablet_service.cc:1505] Processing CreateTablet for tablet c6fb62879dd443a18eb94c506c7a82a2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251215 08:58:22.622109 21396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6fb62879dd443a18eb94c506c7a82a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.625341 21398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d1d9a9be078441587cf5da752d59c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.620736 21405 tablet_service.cc:1505] Processing CreateTablet for tablet b83855234bda47a1b06c3f932728766b (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251215 08:58:22.630203 21405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b83855234bda47a1b06c3f932728766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.634200 21458 tablet_bootstrap.cc:654] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.635833 21249 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1c639a483f4827997eae6f3a6e375d (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251215 08:58:22.649309 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da883c18eeb4679974472b9b341b5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.651842 21323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d0087ada21421eacebc646b271e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.654901 21250 tablet_service.cc:1505] Processing CreateTablet for tablet 1db12563817b44429a055b7dcdd066c3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251215 08:58:22.656276 21250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db12563817b44429a055b7dcdd066c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.656008 21248 tablet_service.cc:1505] Processing CreateTablet for tablet 0351e3362cb742aa87ffc4bf31ceea2c (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251215 08:58:22.664168 21404 tablet_service.cc:1505] Processing CreateTablet for tablet a4e724e065b349faafa05321d2582972 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251215 08:58:22.665478 21404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e724e065b349faafa05321d2582972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.682674 21401 tablet_service.cc:1505] Processing CreateTablet for tablet f4ed801d85bc47d7ac87033f0799bad2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251215 08:58:22.683178 21400 tablet_service.cc:1505] Processing CreateTablet for tablet d20229ce83d84c8ba85a5fc4604d817d (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251215 08:58:22.684046 21401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4ed801d85bc47d7ac87033f0799bad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.688506 21248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0351e3362cb742aa87ffc4bf31ceea2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.688773 21402 tablet_service.cc:1505] Processing CreateTablet for tablet 3b85e43f77684deab6a05166ac30cc14 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251215 08:58:22.691213 21324 tablet_service.cc:1505] Processing CreateTablet for tablet d69415d618114174b070bbc19a876119 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251215 08:58:22.695514 21252 tablet_service.cc:1505] Processing CreateTablet for tablet 124c9efc80174ca585aaee7e0f8a12f0 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251215 08:58:22.689556 21456 tablet_bootstrap.cc:654] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.696723 21256 tablet_service.cc:1505] Processing CreateTablet for tablet 2c10c3de775846868774c98cc0a75094 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251215 08:58:22.697377 21253 tablet_service.cc:1505] Processing CreateTablet for tablet 7da883c18eeb4679974472b9b341b5e8 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251215 08:58:22.703939 21467 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:22.709908 21467 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.710947 21255 tablet_service.cc:1505] Processing CreateTablet for tablet 84bdfb60f8fa4b39bc8cbd437a0809f2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251215 08:58:22.713137 21249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1c639a483f4827997eae6f3a6e375d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.714233 21467 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:22.715242 21252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 124c9efc80174ca585aaee7e0f8a12f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.707350 21247 tablet_service.cc:1505] Processing CreateTablet for tablet d69415d618114174b070bbc19a876119 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251215 08:58:22.698218 21324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69415d618114174b070bbc19a876119. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.698297 21254 tablet_service.cc:1505] Processing CreateTablet for tablet 9582bb1875204089bca2e38a3ff426b3 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251215 08:58:22.730368 21322 tablet_service.cc:1505] Processing CreateTablet for tablet 84bdfb60f8fa4b39bc8cbd437a0809f2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251215 08:58:22.727633 21400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20229ce83d84c8ba85a5fc4604d817d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.751710 21402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b85e43f77684deab6a05166ac30cc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.753226 21330 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf7b0165c00477b9c1b94c6a89052a6 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251215 08:58:22.753806 21255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bdfb60f8fa4b39bc8cbd437a0809f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.753464 21325 tablet_service.cc:1505] Processing CreateTablet for tablet 4d1d9a9be078441587cf5da752d59c33 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251215 08:58:22.755517 21322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bdfb60f8fa4b39bc8cbd437a0809f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.756290 21328 tablet_service.cc:1505] Processing CreateTablet for tablet b83855234bda47a1b06c3f932728766b (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251215 08:58:22.765170 21458 tablet_bootstrap.cc:492] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:22.765662 21458 ts_tablet_manager.cc:1403] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.373s user 0.018s sys 0.009s
W20251215 08:58:22.767442 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:22.771111 21457 tablet_bootstrap.cc:492] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:22.771688 21457 ts_tablet_manager.cc:1403] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.339s user 0.014s sys 0.007s
I20251215 08:58:22.773473 21254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9582bb1875204089bca2e38a3ff426b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.774890 21253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da883c18eeb4679974472b9b341b5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.775635 21256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c10c3de775846868774c98cc0a75094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.783222 21457 raft_consensus.cc:359] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.784358 21247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69415d618114174b070bbc19a876119. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.798218 21457 raft_consensus.cc:385] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:22.798614 21457 raft_consensus.cc:740] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.799497 21457 consensus_queue.cc:260] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.789258 21458 raft_consensus.cc:359] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.803176 21458 raft_consensus.cc:385] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:22.797750 21456 tablet_bootstrap.cc:492] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:22.803499 21458 raft_consensus.cc:740] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.803920 21456 ts_tablet_manager.cc:1403] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.209s user 0.015s sys 0.005s
I20251215 08:58:22.815560 21458 consensus_queue.cc:260] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.818867 21323 tablet_service.cc:1505] Processing CreateTablet for tablet c6fb62879dd443a18eb94c506c7a82a2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251215 08:58:22.824599 21458 ts_tablet_manager.cc:1434] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20251215 08:58:22.827849 21325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d1d9a9be078441587cf5da752d59c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.829636 21330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf7b0165c00477b9c1b94c6a89052a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.833659 21252 tablet_service.cc:1505] Processing CreateTablet for tablet 4c50acc4d0854bd08c6afc283d478ca7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251215 08:58:22.835158 21457 ts_tablet_manager.cc:1434] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.063s user 0.008s sys 0.001s
I20251215 08:58:22.841920 21457 tablet_bootstrap.cc:492] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:22.842343 21326 tablet_service.cc:1505] Processing CreateTablet for tablet f4ed801d85bc47d7ac87033f0799bad2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251215 08:58:22.845503 21456 raft_consensus.cc:359] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.846333 21456 raft_consensus.cc:385] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:22.846619 21456 raft_consensus.cc:740] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:22.758699 21327 tablet_service.cc:1505] Processing CreateTablet for tablet 4c50acc4d0854bd08c6afc283d478ca7 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251215 08:58:22.834996 21252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c50acc4d0854bd08c6afc283d478ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.836185 21329 tablet_service.cc:1505] Processing CreateTablet for tablet a4e724e065b349faafa05321d2582972 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251215 08:58:22.837518 21458 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:22.837581 21328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b83855234bda47a1b06c3f932728766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.859611 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e724e065b349faafa05321d2582972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.861320 21463 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:22.861855 21463 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.863793 21463 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:22.864590 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:22.864866 21323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6fb62879dd443a18eb94c506c7a82a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.866253 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:22.867602 21379 leader_election.cc:343] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:22.868691 21381 leader_election.cc:343] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.869128 21381 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:22.870210 21463 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:22.870821 21249 tablet_service.cc:1505] Processing CreateTablet for tablet 4d1d9a9be078441587cf5da752d59c33 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251215 08:58:22.874931 21326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4ed801d85bc47d7ac87033f0799bad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.876332 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c50acc4d0854bd08c6afc283d478ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.876662 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:22.882112 21324 tablet_service.cc:1505] Processing CreateTablet for tablet 3b85e43f77684deab6a05166ac30cc14 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251215 08:58:22.883409 21324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b85e43f77684deab6a05166ac30cc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.887209 21256 tablet_service.cc:1505] Processing CreateTablet for tablet b83855234bda47a1b06c3f932728766b (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
W20251215 08:58:22.888417 21232 leader_election.cc:343] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.890239 21249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d1d9a9be078441587cf5da752d59c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.894413 21253 tablet_service.cc:1505] Processing CreateTablet for tablet c6fb62879dd443a18eb94c506c7a82a2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251215 08:58:22.895730 21253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6fb62879dd443a18eb94c506c7a82a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.897356 21456 consensus_queue.cc:260] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:22.899780 21456 ts_tablet_manager.cc:1434] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.086s user 0.005s sys 0.000s
I20251215 08:58:22.909374 21256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b83855234bda47a1b06c3f932728766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.909875 21251 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf7b0165c00477b9c1b94c6a89052a6 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251215 08:58:22.911155 21251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf7b0165c00477b9c1b94c6a89052a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.872514 21331 tablet_service.cc:1505] Processing CreateTablet for tablet d20229ce83d84c8ba85a5fc4604d817d (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251215 08:58:22.912387 21456 tablet_bootstrap.cc:492] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:22.913564 21331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20229ce83d84c8ba85a5fc4604d817d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.918576 21248 tablet_service.cc:1505] Processing CreateTablet for tablet a4e724e065b349faafa05321d2582972 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251215 08:58:22.920074 21248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e724e065b349faafa05321d2582972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.923032 21457 tablet_bootstrap.cc:654] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.961871 21255 tablet_service.cc:1505] Processing CreateTablet for tablet d20229ce83d84c8ba85a5fc4604d817d (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251215 08:58:22.963305 21255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20229ce83d84c8ba85a5fc4604d817d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.965513 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:22.967103 21230 leader_election.cc:343] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:22.967583 21230 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:22.968393 21467 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:22.970160 21458 tablet_bootstrap.cc:654] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.928485 21247 tablet_service.cc:1505] Processing CreateTablet for tablet f4ed801d85bc47d7ac87033f0799bad2 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251215 08:58:22.973711 21456 tablet_bootstrap.cc:654] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:22.976748 21250 tablet_service.cc:1505] Processing CreateTablet for tablet 3b85e43f77684deab6a05166ac30cc14 (DEFAULT_TABLE table=test-workload [id=e4317e71ee4e47f08ded586850d0d416]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251215 08:58:22.974572 21247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4ed801d85bc47d7ac87033f0799bad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:22.990020 21250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b85e43f77684deab6a05166ac30cc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:23.020290 21457 tablet_bootstrap.cc:492] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.020833 21457 ts_tablet_manager.cc:1403] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.179s user 0.012s sys 0.002s
I20251215 08:58:23.024623 21458 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.025169 21458 ts_tablet_manager.cc:1403] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.188s user 0.025s sys 0.003s
I20251215 08:58:23.026602 21457 raft_consensus.cc:359] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.027390 21457 raft_consensus.cc:385] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.027684 21457 raft_consensus.cc:740] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.028247 21457 consensus_queue.cc:260] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.030323 21457 ts_tablet_manager.cc:1434] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251215 08:58:23.031342 21457 tablet_bootstrap.cc:492] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.031086 21458 raft_consensus.cc:359] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.031817 21458 raft_consensus.cc:385] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.032119 21458 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.036002 21456 tablet_bootstrap.cc:492] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.036520 21456 ts_tablet_manager.cc:1403] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.124s user 0.017s sys 0.008s
I20251215 08:58:23.038558 21457 tablet_bootstrap.cc:654] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.040650 21456 raft_consensus.cc:359] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.041746 21456 raft_consensus.cc:385] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.042089 21456 raft_consensus.cc:740] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.032801 21458 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.043017 21456 consensus_queue.cc:260] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.044301 21458 ts_tablet_manager.cc:1434] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20251215 08:58:23.045288 21458 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.052719 21467 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.053215 21467 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.055260 21467 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.056391 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.056919 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:23.057801 21230 leader_election.cc:343] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:23.058285 21232 leader_election.cc:343] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.058706 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.059417 21467 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:23.052738 21456 ts_tablet_manager.cc:1434] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.016s user 0.000s sys 0.005s
I20251215 08:58:23.061081 21456 tablet_bootstrap.cc:492] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.070874 21458 tablet_bootstrap.cc:654] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.071674 21456 tablet_bootstrap.cc:654] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.073697 21457 tablet_bootstrap.cc:492] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.074206 21457 ts_tablet_manager.cc:1403] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.001s
I20251215 08:58:23.075366 21458 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.075752 21458 ts_tablet_manager.cc:1403] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20251215 08:58:23.077627 21457 raft_consensus.cc:359] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.078037 21458 raft_consensus.cc:359] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.078290 21457 raft_consensus.cc:385] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.078639 21458 raft_consensus.cc:385] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.078681 21457 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.079034 21458 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.079216 21456 tablet_bootstrap.cc:492] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.079687 21456 ts_tablet_manager.cc:1403] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.001s
I20251215 08:58:23.079495 21457 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.079720 21458 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.081997 21458 ts_tablet_manager.cc:1434] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251215 08:58:23.081998 21457 ts_tablet_manager.cc:1434] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251215 08:58:23.082326 21456 raft_consensus.cc:359] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.083061 21458 tablet_bootstrap.cc:492] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.083035 21456 raft_consensus.cc:385] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.083464 21456 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.084822 21457 tablet_bootstrap.cc:492] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.089352 21458 tablet_bootstrap.cc:654] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.091173 21457 tablet_bootstrap.cc:654] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.091778 21456 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.098656 21456 ts_tablet_manager.cc:1434] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.019s user 0.007s sys 0.001s
I20251215 08:58:23.100044 21456 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.100672 21458 tablet_bootstrap.cc:492] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.101182 21458 ts_tablet_manager.cc:1403] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20251215 08:58:23.103883 21458 raft_consensus.cc:359] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.105145 21458 raft_consensus.cc:385] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.105455 21458 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.106268 21458 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.108565 21456 tablet_bootstrap.cc:654] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.110105 21458 ts_tablet_manager.cc:1434] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251215 08:58:23.110410 21457 tablet_bootstrap.cc:492] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.110991 21457 ts_tablet_manager.cc:1403] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.016s
I20251215 08:58:23.111088 21458 tablet_bootstrap.cc:492] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.115018 21457 raft_consensus.cc:359] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.116837 21457 raft_consensus.cc:385] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.117293 21457 raft_consensus.cc:740] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.117688 21458 tablet_bootstrap.cc:654] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.118485 21457 consensus_queue.cc:260] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.121397 21457 ts_tablet_manager.cc:1434] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20251215 08:58:23.122711 21457 tablet_bootstrap.cc:492] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.124547 21456 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.125092 21456 ts_tablet_manager.cc:1403] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.003s
I20251215 08:58:23.127272 21458 tablet_bootstrap.cc:492] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.127717 21458 ts_tablet_manager.cc:1403] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20251215 08:58:23.129062 21457 tablet_bootstrap.cc:654] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.129918 21458 raft_consensus.cc:359] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.130658 21458 raft_consensus.cc:385] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.130950 21458 raft_consensus.cc:740] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.130882 21456 raft_consensus.cc:359] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.131713 21456 raft_consensus.cc:385] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.131794 21458 consensus_queue.cc:260] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.132158 21456 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.133337 21456 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.134256 21458 ts_tablet_manager.cc:1434] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251215 08:58:23.136137 21458 tablet_bootstrap.cc:492] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.142431 21456 ts_tablet_manager.cc:1434] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20251215 08:58:23.143857 21456 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.142596 21458 tablet_bootstrap.cc:654] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.152405 21456 tablet_bootstrap.cc:654] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.152522 21457 tablet_bootstrap.cc:492] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.153084 21457 ts_tablet_manager.cc:1403] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.003s
I20251215 08:58:23.156564 21457 raft_consensus.cc:359] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.157100 21457 raft_consensus.cc:385] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.157322 21457 raft_consensus.cc:740] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.157755 21456 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.158111 21456 ts_tablet_manager.cc:1403] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20251215 08:58:23.157883 21457 consensus_queue.cc:260] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.160029 21457 ts_tablet_manager.cc:1434] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251215 08:58:23.160336 21456 raft_consensus.cc:359] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.160861 21456 raft_consensus.cc:385] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.161130 21457 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.161167 21456 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.161998 21456 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.164306 21456 ts_tablet_manager.cc:1434] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:58:23.165331 21456 tablet_bootstrap.cc:492] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.167618 21457 tablet_bootstrap.cc:654] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.171669 21458 tablet_bootstrap.cc:492] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.172049 21456 tablet_bootstrap.cc:654] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.172514 21458 ts_tablet_manager.cc:1403] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.010s
I20251215 08:58:23.176226 21458 raft_consensus.cc:359] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.177109 21458 raft_consensus.cc:385] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.177451 21458 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.178156 21457 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.178628 21457 ts_tablet_manager.cc:1403] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20251215 08:58:23.178260 21458 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.181181 21457 raft_consensus.cc:359] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.181806 21457 raft_consensus.cc:385] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.182060 21457 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.182847 21457 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.187644 21457 ts_tablet_manager.cc:1434] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251215 08:58:23.184682 21458 ts_tablet_manager.cc:1434] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20251215 08:58:23.188607 21457 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.189080 21458 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.189854 21475 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.190451 21475 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.194476 21457 tablet_bootstrap.cc:654] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.194489 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:23.196203 21307 leader_election.cc:343] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.196899 21463 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.197429 21463 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.198510 21475 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.199294 21463 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.200555 21458 tablet_bootstrap.cc:654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.202363 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.203006 21265 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:23.203778 21456 tablet_bootstrap.cc:492] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.203488 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.204159 21456 ts_tablet_manager.cc:1403] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.013s
I20251215 08:58:23.204327 21414 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:23.205734 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.206512 21456 raft_consensus.cc:359] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.207065 21475 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.207746 21475 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.208956 21475 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.205649 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.209781 21340 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:23.211452 21379 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:23.212644 21463 raft_consensus.cc:2804] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.213059 21463 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.213148 21467 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.213428 21463 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.213583 21467 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.215618 21467 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.207168 21456 raft_consensus.cc:385] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.217747 21456 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.223456 21467 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.224120 21467 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.223357 21456 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.226586 21467 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.219522 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.228608 21414 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.228843 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.232227 21475 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.233868 21232 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:58:23.240118 21232 leader_election.cc:343] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.241134 21467 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.241530 21467 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.241896 21467 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.243048 21475 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.246142 21463 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.248968 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:23.237161 21499 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.238555 21456 ts_tablet_manager.cc:1434] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.034s user 0.011s sys 0.000s
I20251215 08:58:23.249567 21265 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.249852 21499 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.250878 21456 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.249199 21467 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.253842 21499 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.257470 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:23.258195 21415 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.258108 21265 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:23.258841 21456 tablet_bootstrap.cc:654] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.259800 21381 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.268191 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.268815 21340 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.268710 21415 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:23.269984 21232 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.279403 21340 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:23.280313 21463 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.283471 21500 raft_consensus.cc:2804] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.287524 21500 raft_consensus.cc:697] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:23.288851 21500 consensus_queue.cc:237] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.291774 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:23.292376 21415 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.292745 21458 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.293269 21458 ts_tablet_manager.cc:1403] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.104s user 0.044s sys 0.049s
I20251215 08:58:23.295326 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.308415 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.307762 21458 raft_consensus.cc:359] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.299624 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.308991 21414 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:23.300562 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:23.308964 21458 raft_consensus.cc:385] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.309679 21341 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.296808 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.310210 21458 raft_consensus.cc:740] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.295787 21467 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.310715 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.298506 21456 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.307936 21457 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.311770 21230 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.311794 21458 consensus_queue.cc:260] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.314105 21457 ts_tablet_manager.cc:1403] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.126s user 0.030s sys 0.012s
I20251215 08:58:23.314610 21467 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.315110 21467 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.315578 21467 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.306633 21501 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.309873 21503 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.317256 21501 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.313549 21456 ts_tablet_manager.cc:1403] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.063s user 0.016s sys 0.000s
I20251215 08:58:23.319361 21415 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:23.320647 21456 raft_consensus.cc:359] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.321314 21456 raft_consensus.cc:385] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.321614 21456 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.321898 21457 raft_consensus.cc:359] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.322722 21457 raft_consensus.cc:385] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.323045 21457 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.323220 21501 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.324702 21498 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.325428 21500 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.325146 21498 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.306089 21504 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.323390 21163 catalog_manager.cc:5654] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:23.326396 21504 raft_consensus.cc:697] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:58:23.312724 21307 leader_election.cc:343] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.327155 21498 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.323945 21457 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.317822 21503 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.330090 21457 ts_tablet_manager.cc:1434] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
W20251215 08:58:23.327802 21307 leader_election.cc:343] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.330886 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.331056 21463 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.331143 21458 ts_tablet_manager.cc:1434] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.038s user 0.009s sys 0.016s
I20251215 08:58:23.331637 21498 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.331604 21463 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.332080 21498 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.332193 21458 tablet_bootstrap.cc:492] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.332448 21498 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.325852 21500 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.333657 21463 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.334724 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.335320 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.335947 21265 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:23.337328 21381 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.338335 21458 tablet_bootstrap.cc:654] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.338792 21500 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.339078 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.339489 21500 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.339829 21265 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:23.340176 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.327265 21504 consensus_queue.cc:237] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.341157 21381 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.340911 21415 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.342500 21232 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.339831 21500 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.343286 21500 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.343387 21506 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.343457 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.343927 21506 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.344313 21506 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.344487 21463 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.345824 21463 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.346611 21463 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.344980 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.347159 21414 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.344132 21340 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:23.343277 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.350132 21499 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.350633 21499 raft_consensus.cc:697] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:23.322408 21456 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.350574 21232 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.340210 21498 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.351613 21499 consensus_queue.cc:237] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.352152 21501 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.352569 21501 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.352924 21501 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.352867 21339 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:23.353920 21506 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.356722 21506 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.357466 21457 tablet_bootstrap.cc:492] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.357399 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:58:23.358855 21230 leader_election.cc:343] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.344519 21503 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:58:23.360597 21230 leader_election.cc:343] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.362643 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.359951 21463 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.363706 21458 tablet_bootstrap.cc:492] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.363608 21501 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.364166 21458 ts_tablet_manager.cc:1403] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.003s
I20251215 08:58:23.364436 21414 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:23.364748 21265 raft_consensus.cc:2393] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:23.366140 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.366756 21415 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:23.366809 21458 raft_consensus.cc:359] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.367043 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:23.367532 21458 raft_consensus.cc:385] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.367866 21458 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.368085 21265 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:23.368495 21458 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.370195 21498 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.370685 21458 ts_tablet_manager.cc:1434] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251215 08:58:23.371753 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.371873 21458 tablet_bootstrap.cc:492] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.372030 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.372366 21339 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.373739 21379 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.374049 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.374624 21339 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.376334 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.376856 21339 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.376868 21463 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.377940 21463 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251215 08:58:23.379395 21501 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.379993 21458 tablet_bootstrap.cc:654] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.380877 21457 tablet_bootstrap.cc:654] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.382932 21456 ts_tablet_manager.cc:1434] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20251215 08:58:23.383987 21456 tablet_bootstrap.cc:492] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.385771 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:23.392701 21339 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:23.393391 21457 tablet_bootstrap.cc:492] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.393852 21457 ts_tablet_manager.cc:1403] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.005s
I20251215 08:58:23.394331 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:23.396412 21457 raft_consensus.cc:359] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.397053 21457 raft_consensus.cc:385] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.397329 21457 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.397491 21500 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.398032 21457 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.400372 21457 ts_tablet_manager.cc:1434] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20251215 08:58:23.401346 21457 tablet_bootstrap.cc:492] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.401127 21499 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.402005 21499 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.400076 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:23.402456 21499 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.403066 21265 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.390343 21467 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.405452 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:23.405979 21414 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.403061 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:23.408789 21415 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.409919 21458 tablet_bootstrap.cc:492] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.410370 21458 ts_tablet_manager.cc:1403] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.000s
I20251215 08:58:23.410300 21265 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:23.408131 21457 tablet_bootstrap.cc:654] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.411603 21381 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.412655 21463 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.413228 21463 raft_consensus.cc:697] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:23.413959 21456 tablet_bootstrap.cc:654] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.413805 21458 raft_consensus.cc:359] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.414026 21463 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.414554 21458 raft_consensus.cc:385] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.417337 21458 raft_consensus.cc:740] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.417977 21458 consensus_queue.cc:260] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.417230 21414 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:23.420147 21467 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.420780 21232 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.421420 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:58:23.402295 21307 leader_election.cc:343] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:23.425546 21232 leader_election.cc:343] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.425827 21458 ts_tablet_manager.cc:1434] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20251215 08:58:23.426218 21467 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.426543 21415 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:23.426888 21458 tablet_bootstrap.cc:492] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.427031 21467 raft_consensus.cc:697] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:23.428339 21467 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.429637 21499 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.429425 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.430555 21339 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.431504 21499 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.433117 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.433842 21340 raft_consensus.cc:2393] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:23.432636 21500 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.439324 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.439913 21341 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.454838 21458 tablet_bootstrap.cc:654] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.455847 21230 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.457719 21456 tablet_bootstrap.cc:492] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.458187 21456 ts_tablet_manager.cc:1403] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.074s user 0.016s sys 0.021s
I20251215 08:58:23.458748 21504 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.460520 21504 raft_consensus.cc:697] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:23.461362 21504 consensus_queue.cc:237] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.462805 21457 tablet_bootstrap.cc:492] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.463266 21457 ts_tablet_manager.cc:1403] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.013s
I20251215 08:58:23.467432 21339 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:23.468284 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:23.468897 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.469086 21415 raft_consensus.cc:2393] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:23.466396 21341 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:23.491710 21467 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.504634 21161 catalog_manager.cc:5654] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.513437 21463 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.467628 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:23.516979 21501 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.526535 21467 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.513345 21499 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.527118 21501 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.511693 21164 catalog_manager.cc:5654] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.527318 21457 raft_consensus.cc:359] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.531183 21499 raft_consensus.cc:697] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:23.531903 21499 consensus_queue.cc:237] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.530642 21456 raft_consensus.cc:359] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.534315 21456 raft_consensus.cc:385] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.534519 21456 raft_consensus.cc:740] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.535032 21456 consensus_queue.cc:260] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.535832 21457 raft_consensus.cc:385] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.536073 21457 raft_consensus.cc:740] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.527786 21463 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.536782 21457 consensus_queue.cc:260] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.538089 21456 ts_tablet_manager.cc:1434] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.080s user 0.005s sys 0.001s
I20251215 08:58:23.539686 21456 tablet_bootstrap.cc:492] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.539808 21501 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.540845 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:23.542064 21232 leader_election.cc:343] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.542268 21463 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.542892 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.543555 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.544085 21467 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:23.544211 21381 leader_election.cc:343] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:23.544812 21379 leader_election.cc:343] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.545251 21379 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.545770 21457 ts_tablet_manager.cc:1434] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.082s user 0.004s sys 0.003s
I20251215 08:58:23.547116 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.547770 21414 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.548929 21232 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.549986 21467 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.550208 21457 tablet_bootstrap.cc:492] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.550390 21467 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.550936 21463 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:23.551565 21467 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:23.530215 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 57606.2us
I20251215 08:58:23.553709 21456 tablet_bootstrap.cc:654] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.554662 21230 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.556627 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.557265 21341 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.557827 21501 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.558200 21501 raft_consensus.cc:697] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:23.558816 21501 consensus_queue.cc:237] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.562835 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.563059 21230 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.563184 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.563896 21501 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.564778 21501 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.564563 21499 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251215 08:58:23.567553 21230 leader_election.cc:343] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.568746 21501 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.572518 21458 tablet_bootstrap.cc:492] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.573153 21458 ts_tablet_manager.cc:1403] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.146s user 0.015s sys 0.023s
I20251215 08:58:23.576746 21458 raft_consensus.cc:359] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.577260 21458 raft_consensus.cc:385] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.577477 21458 raft_consensus.cc:740] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.578080 21458 consensus_queue.cc:260] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.579704 21458 ts_tablet_manager.cc:1434] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251215 08:58:23.580356 21458 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.581418 21457 tablet_bootstrap.cc:654] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.583614 21467 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.587754 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:23.588289 21415 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.588949 21458 tablet_bootstrap.cc:654] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.589655 21501 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.590920 21456 tablet_bootstrap.cc:492] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.591351 21456 ts_tablet_manager.cc:1403] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.013s
I20251215 08:58:23.593086 21499 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.593545 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.593709 21499 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.596300 21499 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.597134 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.598304 21463 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:58:23.598670 21307 leader_election.cc:343] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.598743 21463 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.600831 21463 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.601262 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.601986 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.603142 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:23.603760 21381 leader_election.cc:343] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:23.604688 21379 leader_election.cc:343] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.605127 21467 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.605814 21379 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:58:23.606801 21307 leader_election.cc:343] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.607221 21307 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.608440 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:58:23.611976 21232 leader_election.cc:343] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.612182 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.612771 21339 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.610041 21499 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251215 08:58:23.616001 21230 leader_election.cc:343] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.609478 21463 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:23.609498 21501 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.619840 21456 raft_consensus.cc:359] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.620481 21456 raft_consensus.cc:385] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.620729 21456 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.621464 21456 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.623644 21456 ts_tablet_manager.cc:1434] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.032s user 0.000s sys 0.005s
I20251215 08:58:23.625933 21456 tablet_bootstrap.cc:492] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.628063 21499 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.628499 21499 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.629328 21415 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:23.629359 21339 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:23.631158 21230 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.632221 21501 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.632676 21501 raft_consensus.cc:697] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:23.633402 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.633484 21501 consensus_queue.cc:237] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.634290 21415 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:23.635375 21499 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.635859 21456 tablet_bootstrap.cc:654] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.636722 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.637506 21265 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:23.638768 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:23.640694 21232 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:23.641857 21467 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.642489 21467 raft_consensus.cc:697] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:23.643576 21467 consensus_queue.cc:237] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:23.644778 21457 tablet_bootstrap.cc:492] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.645412 21457 ts_tablet_manager.cc:1403] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.099s user 0.042s sys 0.008s
I20251215 08:58:23.640906 21164 catalog_manager.cc:5654] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.648316 21457 raft_consensus.cc:359] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.649181 21457 raft_consensus.cc:385] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.649714 21457 raft_consensus.cc:740] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.650672 21499 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.650727 21457 consensus_queue.cc:260] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.654237 21457 ts_tablet_manager.cc:1434] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20251215 08:58:23.655187 21457 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.659925 21467 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.659196 21163 catalog_manager.cc:5654] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:23.660953 21457 tablet_bootstrap.cc:654] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.661654 21463 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.662089 21463 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.664120 21463 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.665846 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.666539 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:23.667160 21265 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:23.660478 21467 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:23.667217 21379 leader_election.cc:343] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.670033 21456 tablet_bootstrap.cc:492] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.670504 21456 ts_tablet_manager.cc:1403] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.007s
I20251215 08:58:23.673925 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.674007 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.674845 21415 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:23.675971 21456 raft_consensus.cc:359] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.676177 21232 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.676635 21456 raft_consensus.cc:385] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.676944 21456 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.677278 21504 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.669945 21467 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.677572 21456 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.677201 21381 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:23.685314 21456 ts_tablet_manager.cc:1434] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20251215 08:58:23.686277 21456 tablet_bootstrap.cc:492] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.688513 21504 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:23.680378 21161 catalog_manager.cc:5654] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.682770 21463 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.690032 21499 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:23.695439 21458 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.695885 21458 ts_tablet_manager.cc:1403] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.116s user 0.014s sys 0.015s
I20251215 08:58:23.698033 21457 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.698601 21457 ts_tablet_manager.cc:1403] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.009s
I20251215 08:58:23.693691 21463 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.705138 21463 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.705806 21161 catalog_manager.cc:5654] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.708207 21161 catalog_manager.cc:5654] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.727320 21458 raft_consensus.cc:359] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.728003 21458 raft_consensus.cc:385] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.728245 21458 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.728858 21458 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.730892 21458 ts_tablet_manager.cc:1434] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.035s user 0.005s sys 0.001s
I20251215 08:58:23.731768 21458 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.732415 21456 tablet_bootstrap.cc:654] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.724472 21463 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.735646 21463 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:23.736681 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:23.737831 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:58:23.737988 21379 leader_election.cc:343] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:23.738427 21266 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.741477 21458 tablet_bootstrap.cc:654] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.747721 21266 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:23.749027 21463 consensus_queue.cc:1048] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:23.751448 21381 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:23.752530 21463 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:23.752990 21463 raft_consensus.cc:697] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:23.753840 21463 consensus_queue.cc:237] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.758579 21457 raft_consensus.cc:359] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.766165 21457 raft_consensus.cc:385] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.766522 21457 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.767038 21458 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.767654 21458 ts_tablet_manager.cc:1403] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.006s
I20251215 08:58:23.769043 21463 consensus_queue.cc:1048] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251215 08:58:23.769757 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:23.767210 21457 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.770955 21458 raft_consensus.cc:359] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.771814 21458 raft_consensus.cc:385] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.772104 21458 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.775247 21458 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.772212 21457 ts_tablet_manager.cc:1434] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.073s user 0.002s sys 0.003s
I20251215 08:58:23.777405 21457 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.779047 21458 ts_tablet_manager.cc:1434] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20251215 08:58:23.780170 21458 tablet_bootstrap.cc:492] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.785537 21163 catalog_manager.cc:5654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.787851 21458 tablet_bootstrap.cc:654] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.788136 21163 catalog_manager.cc:5654] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:23.794231 21161 catalog_manager.cc:5654] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:23.795490 21457 tablet_bootstrap.cc:654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.813711 21457 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.814177 21457 ts_tablet_manager.cc:1403] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.004s
I20251215 08:58:23.816715 21457 raft_consensus.cc:359] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.821780 21457 raft_consensus.cc:385] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.822063 21457 raft_consensus.cc:740] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.822754 21501 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.823402 21501 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.823894 21457 consensus_queue.cc:260] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.825641 21501 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.827332 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.827572 21456 tablet_bootstrap.cc:492] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.827877 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.828104 21456 ts_tablet_manager.cc:1403] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.142s user 0.027s sys 0.027s
I20251215 08:58:23.828017 21458 tablet_bootstrap.cc:492] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.828609 21458 ts_tablet_manager.cc:1403] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.024s
I20251215 08:58:23.828640 21415 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:23.830647 21456 raft_consensus.cc:359] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.831243 21456 raft_consensus.cc:385] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.831387 21458 raft_consensus.cc:359] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.831507 21232 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:23.832054 21458 raft_consensus.cc:385] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.831518 21456 raft_consensus.cc:740] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.832360 21458 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.832980 21458 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.832980 21456 consensus_queue.cc:260] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.835247 21458 ts_tablet_manager.cc:1434] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:58:23.835459 21456 ts_tablet_manager.cc:1434] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20251215 08:58:23.836064 21458 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.837085 21504 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.838121 21456 tablet_bootstrap.cc:492] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.842267 21458 tablet_bootstrap.cc:654] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.852646 21457 ts_tablet_manager.cc:1434] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20251215 08:58:23.853960 21457 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.859508 21501 consensus_queue.cc:1048] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:23.865861 21504 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:23.860277 21467 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:23.861047 21456 tablet_bootstrap.cc:654] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.861915 21458 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.865193 21457 tablet_bootstrap.cc:654] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.872268 21458 ts_tablet_manager.cc:1403] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.016s
I20251215 08:58:23.873272 21467 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:23.875334 21458 raft_consensus.cc:359] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.876122 21458 raft_consensus.cc:385] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.876482 21458 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.877277 21458 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.880023 21458 ts_tablet_manager.cc:1434] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20251215 08:58:23.883219 21499 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:23.887957 21458 tablet_bootstrap.cc:492] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.894726 21339 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.895013 21467 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:23.895702 21458 tablet_bootstrap.cc:654] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.905356 21467 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:23.915742 21458 tablet_bootstrap.cc:492] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:23.916445 21458 ts_tablet_manager.cc:1403] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.000s
I20251215 08:58:23.919706 21458 raft_consensus.cc:359] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.920558 21458 raft_consensus.cc:385] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.925230 21500 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:23.920889 21458 raft_consensus.cc:740] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.928850 21458 consensus_queue.cc:260] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.930050 21498 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:23.932397 21458 ts_tablet_manager.cc:1434] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20251215 08:58:23.933673 21458 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:23.937108 21456 tablet_bootstrap.cc:492] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:23.937588 21456 ts_tablet_manager.cc:1403] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.100s user 0.022s sys 0.017s
I20251215 08:58:23.938581 21467 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:23.939153 21467 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.940157 21456 raft_consensus.cc:359] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.940765 21456 raft_consensus.cc:385] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.941021 21456 raft_consensus.cc:740] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.941630 21456 consensus_queue.cc:260] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.943149 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:23.943717 21456 ts_tablet_manager.cc:1434] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251215 08:58:23.943773 21337 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.941035 21467 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:23.944907 21456 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:23.945775 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:23.946812 21467 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:23.947235 21467 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:23.947666 21467 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:23.950006 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:23.953948 21456 tablet_bootstrap.cc:654] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.950666 21413 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:23.958802 21458 tablet_bootstrap.cc:654] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:23.959115 21457 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:23.959551 21457 ts_tablet_manager.cc:1403] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.106s user 0.019s sys 0.012s
I20251215 08:58:23.962385 21457 raft_consensus.cc:359] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.963056 21457 raft_consensus.cc:385] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:23.963315 21457 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:23.964226 21457 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:23.967012 21457 ts_tablet_manager.cc:1434] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251215 08:58:23.967850 21457 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:23.976488 21467 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.002292 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.002974 21340 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.006353 21503 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:24.013582 21467 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.015944 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:24.016716 21413 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.029873 21500 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:24.031050 21554 consensus_queue.cc:1048] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:24.031724 21340 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:24.047670 21500 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.043210 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.060024 21504 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.060722 21504 raft_consensus.cc:697] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:24.065373 21554 consensus_queue.cc:1048] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:24.066362 21501 consensus_queue.cc:1048] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251215 08:58:24.079875 21232 consensus_peers.cc:597] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:24.050096 21500 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.075600 21504 consensus_queue.cc:237] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.095598 21456 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.096235 21456 ts_tablet_manager.cc:1403] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.151s user 0.035s sys 0.037s
I20251215 08:58:24.085903 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:24.098791 21381 leader_election.cc:343] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.099603 21456 raft_consensus.cc:359] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.100543 21456 raft_consensus.cc:385] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.100433 21467 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:24.100966 21456 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.102025 21456 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.108114 21456 ts_tablet_manager.cc:1434] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20251215 08:58:24.093257 21500 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.105126 21499 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.118875 21499 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.119436 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.125288 21499 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.127131 21456 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.128500 21457 tablet_bootstrap.cc:654] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.128861 21503 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:58:24.135991 21379 leader_election.cc:343] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.140743 21463 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.161621 21463 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.147207 21379 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.145148 21503 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.166290 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.167004 21266 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:24.145942 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.178782 21561 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.180299 21561 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.170087 21500 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.203195 21498 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.203858 21498 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.207918 21456 tablet_bootstrap.cc:654] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.185949 21413 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:24.216585 21463 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:24.216852 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 83048.1us
W20251215 08:58:24.217346 21307 leader_election.cc:343] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.226210 21463 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.216657 21498 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.222357 21503 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.223376 21561 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.217948 21458 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:24.230324 21458 ts_tablet_manager.cc:1403] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.297s user 0.038s sys 0.062s
I20251215 08:58:24.230430 21511 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.227288 21504 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.227854 21463 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.228520 21501 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:24.231145 21511 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.240229 21511 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.240952 21511 consensus_queue.cc:1048] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:24.244002 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.233664 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:24.234845 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.235677 21504 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.245301 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.246701 21458 raft_consensus.cc:359] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.255625 21458 raft_consensus.cc:385] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.255990 21458 raft_consensus.cc:740] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.256254 21504 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.256783 21458 consensus_queue.cc:260] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.254046 21264 raft_consensus.cc:2393] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:24.259867 21463 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.260457 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.261412 21498 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.261802 21498 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.262115 21498 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.262038 21164 catalog_manager.cc:5654] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:24.263571 21458 ts_tablet_manager.cc:1434] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20251215 08:58:24.264811 21458 tablet_bootstrap.cc:492] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:24.270587 21456 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.271010 21456 ts_tablet_manager.cc:1403] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.144s user 0.023s sys 0.016s
I20251215 08:58:24.274660 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.275249 21264 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:24.277043 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.278486 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.278936 21456 raft_consensus.cc:359] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.281049 21456 raft_consensus.cc:385] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.281102 21337 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:24.281607 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.281889 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.282120 21456 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.282234 21413 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:24.282467 21340 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:24.270345 21498 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.282970 21456 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.284147 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.285279 21498 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.286046 21456 ts_tablet_manager.cc:1434] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20251215 08:58:24.286253 21457 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:24.286720 21457 ts_tablet_manager.cc:1403] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.319s user 0.024s sys 0.011s
I20251215 08:58:24.286939 21456 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.290284 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.291749 21499 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.290964 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.294107 21456 tablet_bootstrap.cc:654] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.291280 21457 raft_consensus.cc:359] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.295779 21457 raft_consensus.cc:385] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.296119 21457 raft_consensus.cc:740] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.296921 21457 consensus_queue.cc:260] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.299695 21457 ts_tablet_manager.cc:1434] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20251215 08:58:24.300591 21499 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.291767 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.300838 21457 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:24.302035 21414 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:24.307281 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:24.307893 21266 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.310259 21458 tablet_bootstrap.cc:654] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:24.304770 21381 leader_election.cc:343] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.305104 21415 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.305514 21379 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.310894 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:24.304529 21230 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.312914 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:58:24.316922 21232 leader_election.cc:343] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.319229 21511 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.319626 21511 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.319936 21511 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.323575 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.324998 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.325161 21337 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:24.325655 21340 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:24.325989 21503 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.326339 21503 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.327455 21503 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.332846 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.334059 21561 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.334439 21561 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.334746 21561 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.344277 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.348685 21503 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.349705 21463 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.350144 21463 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.350482 21463 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.351020 21503 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.353991 21500 consensus_queue.cc:1048] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:24.355080 21561 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:24.338764 21307 leader_election.cc:343] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.358029 21561 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:58:24.335722 21230 leader_election.cc:343] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.359346 21230 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:58:24.360431 21230 leader_election.cc:343] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.363231 21568 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.366942 21266 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:24.367973 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:24.368764 21498 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.368824 21463 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.369330 21498 raft_consensus.cc:697] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:24.370159 21498 consensus_queue.cc:237] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.370783 21463 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.373474 21503 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:24.374610 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:24.352125 21511 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:24.377741 21381 leader_election.cc:343] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.382031 21457 tablet_bootstrap.cc:654] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.384214 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:24.385208 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:24.385139 21503 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:24.362543 21561 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.393122 21511 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.385807 21413 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.394632 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:24.395720 21264 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:24.390465 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.405730 21413 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:24.406445 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.407203 21340 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.401575 21338 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.416906 21415 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:24.456971 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.458766 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.460069 21561 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.460714 21561 raft_consensus.cc:697] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:24.461571 21561 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.463403 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.485484 21379 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.487296 21573 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.491732 21338 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:24.495344 21340 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:24.514429 21458 tablet_bootstrap.cc:492] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:24.515024 21458 ts_tablet_manager.cc:1403] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.250s user 0.052s sys 0.072s
I20251215 08:58:24.516144 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.516989 21456 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.517467 21456 ts_tablet_manager.cc:1403] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.231s user 0.013s sys 0.012s
I20251215 08:58:24.523962 21456 raft_consensus.cc:359] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.518360 21573 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.524484 21163 catalog_manager.cc:5654] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:24.525688 21573 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:24.524884 21456 raft_consensus.cc:385] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.526710 21456 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.528282 21456 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:24.510006 21230 leader_election.cc:343] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.531279 21573 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.534399 21456 ts_tablet_manager.cc:1434] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20251215 08:58:24.535413 21456 tablet_bootstrap.cc:492] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.556236 21230 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:24.558269 21561 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.558779 21561 raft_consensus.cc:697] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:24.559712 21561 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.569298 21501 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:24.573521 21457 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:24.574010 21457 ts_tablet_manager.cc:1403] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.273s user 0.012s sys 0.009s
I20251215 08:58:24.578970 21458 raft_consensus.cc:359] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.579730 21458 raft_consensus.cc:385] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.580720 21458 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.581962 21458 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.583983 21457 raft_consensus.cc:359] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.584924 21457 raft_consensus.cc:385] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.585325 21457 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.588917 21467 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251215 08:58:24.605569 21498 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.606233 21498 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.608696 21498 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.613694 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.616529 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.617408 21264 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:24.621652 21163 catalog_manager.cc:5654] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:24.623163 21161 catalog_manager.cc:5654] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:24.625020 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.630507 21498 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.630868 21498 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:58:24.630720 21307 leader_election.cc:343] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.631369 21498 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.638475 21498 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.640826 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:24.641362 21264 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.641716 21458 ts_tablet_manager.cc:1434] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.126s user 0.024s sys 0.029s
I20251215 08:58:24.642694 21458 tablet_bootstrap.cc:492] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:24.645182 21457 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.645864 21499 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.646274 21499 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.647174 21573 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.647302 21457 ts_tablet_manager.cc:1434] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.073s user 0.006s sys 0.000s
I20251215 08:58:24.647624 21573 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.648151 21499 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.648859 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:24.650590 21307 leader_election.cc:343] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.651130 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.651479 21457 tablet_bootstrap.cc:492] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:24.651572 21456 tablet_bootstrap.cc:654] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.651722 21415 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:24.654786 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.655354 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.655534 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.652951 21573 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.656281 21340 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
W20251215 08:58:24.656458 21381 leader_election.cc:343] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.657037 21498 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.660290 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.661167 21573 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.661711 21573 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.662248 21573 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.659940 21499 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.652417 21264 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:24.667047 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:24.671020 21499 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.671667 21499 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.673434 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.686455 21498 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.688251 21498 raft_consensus.cc:697] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:24.689944 21498 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.690645 21573 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.696101 21573 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.695487 21457 tablet_bootstrap.cc:654] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:24.692206 21307 leader_election.cc:343] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.697917 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.697718 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:24.698397 21499 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.699518 21340 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:24.687368 21162 catalog_manager.cc:5654] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:24.700664 21499 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.702178 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:24.703118 21458 tablet_bootstrap.cc:654] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:24.703682 21381 leader_election.cc:343] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.704246 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:24.704715 21381 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.705148 21413 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:24.705754 21511 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251215 08:58:24.703673 21307 leader_election.cc:343] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.710299 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.713162 21456 tablet_bootstrap.cc:492] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.713656 21456 ts_tablet_manager.cc:1403] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.178s user 0.010s sys 0.014s
I20251215 08:58:24.714660 21499 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.719700 21504 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.720224 21504 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.722470 21504 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.716429 21456 raft_consensus.cc:359] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.724234 21456 raft_consensus.cc:385] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.724376 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.724570 21456 raft_consensus.cc:740] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.724968 21413 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:24.725419 21456 consensus_queue.cc:260] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.727700 21232 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.728034 21456 ts_tablet_manager.cc:1434] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20251215 08:58:24.728335 21501 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.728685 21501 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.728987 21501 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.728936 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.728914 21456 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.731424 21162 catalog_manager.cc:5654] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:24.741086 21457 tablet_bootstrap.cc:492] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:24.741248 21456 tablet_bootstrap.cc:654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.741524 21457 ts_tablet_manager.cc:1403] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.090s user 0.018s sys 0.002s
W20251215 08:58:24.741963 21230 leader_election.cc:343] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.744140 21457 raft_consensus.cc:359] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.744868 21457 raft_consensus.cc:385] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.745234 21457 raft_consensus.cc:740] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.745837 21457 consensus_queue.cc:260] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.746762 21501 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.747141 21456 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.747669 21456 ts_tablet_manager.cc:1403] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20251215 08:58:24.749624 21501 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.749848 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:24.750362 21413 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.750972 21457 ts_tablet_manager.cc:1434] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251215 08:58:24.751183 21456 raft_consensus.cc:359] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.751784 21456 raft_consensus.cc:385] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.752013 21457 tablet_bootstrap.cc:492] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:24.752054 21456 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.752830 21456 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.754807 21458 tablet_bootstrap.cc:492] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:24.755214 21458 ts_tablet_manager.cc:1403] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.113s user 0.005s sys 0.017s
I20251215 08:58:24.756877 21456 ts_tablet_manager.cc:1434] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251215 08:58:24.757298 21413 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:24.757956 21456 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.758334 21232 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:24.759128 21501 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.759583 21501 raft_consensus.cc:697] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:24.760334 21501 consensus_queue.cc:237] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.761323 21458 raft_consensus.cc:359] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.761976 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.761996 21458 raft_consensus.cc:385] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.763152 21458 raft_consensus.cc:740] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.763788 21458 consensus_queue.cc:260] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.764890 21456 tablet_bootstrap.cc:654] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.766187 21458 ts_tablet_manager.cc:1434] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20251215 08:58:24.767362 21458 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
W20251215 08:58:24.770394 21230 leader_election.cc:343] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.777369 21456 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.777776 21456 ts_tablet_manager.cc:1403] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.005s
I20251215 08:58:24.779453 21162 catalog_manager.cc:5654] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:24.782904 21458 tablet_bootstrap.cc:654] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.785456 21457 tablet_bootstrap.cc:654] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.787374 21456 raft_consensus.cc:359] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.787998 21456 raft_consensus.cc:385] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.788241 21456 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.788949 21456 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.790863 21456 ts_tablet_manager.cc:1434] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20251215 08:58:24.791913 21456 tablet_bootstrap.cc:492] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.798444 21458 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:24.798871 21458 ts_tablet_manager.cc:1403] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.004s
I20251215 08:58:24.801494 21456 tablet_bootstrap.cc:654] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.801167 21458 raft_consensus.cc:359] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.801904 21458 raft_consensus.cc:385] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.802170 21458 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.802840 21458 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.810029 21457 tablet_bootstrap.cc:492] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:24.810644 21457 ts_tablet_manager.cc:1403] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.000s
I20251215 08:58:24.814348 21458 ts_tablet_manager.cc:1434] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
W20251215 08:58:24.814592 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:24.815207 21458 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:24.815902 21457 raft_consensus.cc:359] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.817049 21457 raft_consensus.cc:385] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.817365 21457 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.818110 21457 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.820905 21458 tablet_bootstrap.cc:654] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.825510 21457 ts_tablet_manager.cc:1434] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20251215 08:58:24.826479 21457 tablet_bootstrap.cc:492] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
W20251215 08:58:24.833382 21307 consensus_peers.cc:597] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:24.848122 21498 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.848462 21337 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.848685 21498 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.849452 21457 tablet_bootstrap.cc:654] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.850575 21456 tablet_bootstrap.cc:492] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.851167 21456 ts_tablet_manager.cc:1403] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.012s
I20251215 08:58:24.853084 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:24.854399 21307 leader_election.cc:343] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.851222 21498 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.854554 21456 raft_consensus.cc:359] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.855602 21456 raft_consensus.cc:385] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.855693 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.855901 21456 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
W20251215 08:58:24.857156 21307 leader_election.cc:343] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.856987 21456 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.857700 21307 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.858345 21458 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:24.858810 21458 ts_tablet_manager.cc:1403] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.004s
I20251215 08:58:24.858919 21457 tablet_bootstrap.cc:492] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:24.859338 21457 ts_tablet_manager.cc:1403] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.003s
I20251215 08:58:24.859580 21456 ts_tablet_manager.cc:1434] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251215 08:58:24.859990 21498 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.856990 21501 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:24.861541 21456 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.861610 21458 raft_consensus.cc:359] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.862504 21458 raft_consensus.cc:385] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.862423 21457 raft_consensus.cc:359] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.862900 21458 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.863094 21457 raft_consensus.cc:385] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.863418 21457 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.863799 21458 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.864137 21457 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.866165 21458 ts_tablet_manager.cc:1434] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:24.867677 21456 tablet_bootstrap.cc:654] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.869932 21457 ts_tablet_manager.cc:1434] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.010s user 0.002s sys 0.005s
I20251215 08:58:24.871093 21457 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:24.873814 21458 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:24.892990 21458 tablet_bootstrap.cc:654] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.895381 21457 tablet_bootstrap.cc:654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.902206 21456 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.902792 21456 ts_tablet_manager.cc:1403] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.015s
I20251215 08:58:24.904304 21467 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:24.905843 21456 raft_consensus.cc:359] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.906698 21456 raft_consensus.cc:385] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.907002 21456 raft_consensus.cc:740] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.907768 21456 consensus_queue.cc:260] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.908895 21511 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.910367 21511 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.912184 21511 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.913679 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:24.914839 21458 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
W20251215 08:58:24.915237 21381 leader_election.cc:343] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.915378 21458 ts_tablet_manager.cc:1403] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.008s
I20251215 08:58:24.918458 21458 raft_consensus.cc:359] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.918989 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.919304 21458 raft_consensus.cc:385] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.919690 21458 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.919785 21337 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:24.923491 21456 ts_tablet_manager.cc:1434] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20251215 08:58:24.920459 21458 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.924712 21504 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.925254 21457 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:24.925118 21504 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.925650 21379 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.926256 21458 ts_tablet_manager.cc:1434] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20251215 08:58:24.926692 21573 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:24.926841 21504 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.927119 21458 tablet_bootstrap.cc:492] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:24.927031 21573 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:24.927455 21573 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.932399 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.932971 21458 tablet_bootstrap.cc:654] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.932775 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:24.933372 21499 consensus_queue.cc:1048] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251215 08:58:24.934144 21230 leader_election.cc:343] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.934120 21573 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:24.934962 21232 leader_election.cc:343] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.935519 21456 tablet_bootstrap.cc:492] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.935340 21232 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:24.937033 21504 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:24.937490 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:58:24.939139 21381 leader_election.cc:343] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:24.925640 21457 ts_tablet_manager.cc:1403] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.019s
I20251215 08:58:24.942482 21573 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:24.945223 21457 raft_consensus.cc:359] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.946110 21457 raft_consensus.cc:385] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.946419 21457 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.950042 21457 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.951257 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:24.952929 21337 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:24.956696 21458 tablet_bootstrap.cc:492] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:24.957100 21456 tablet_bootstrap.cc:654] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.957125 21458 ts_tablet_manager.cc:1403] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.008s
I20251215 08:58:24.960088 21458 raft_consensus.cc:359] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.960742 21458 raft_consensus.cc:385] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.961028 21458 raft_consensus.cc:740] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.953411 21457 ts_tablet_manager.cc:1434] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20251215 08:58:24.962253 21457 tablet_bootstrap.cc:492] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:24.966073 21458 consensus_queue.cc:260] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.971179 21499 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:24.971709 21499 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.972576 21458 ts_tablet_manager.cc:1434] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20251215 08:58:24.974396 21458 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:24.978479 21337 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:24.976492 21499 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:24.980230 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:24.981326 21379 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:24.980996 21415 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:24.982683 21573 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:24.983120 21456 tablet_bootstrap.cc:492] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:24.983337 21457 tablet_bootstrap.cc:654] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.983122 21573 raft_consensus.cc:697] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:24.983732 21456 ts_tablet_manager.cc:1403] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.015s
I20251215 08:58:24.984328 21573 consensus_queue.cc:237] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.986423 21456 raft_consensus.cc:359] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.987125 21456 raft_consensus.cc:385] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:24.987427 21456 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:24.988103 21456 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:24.989290 21458 tablet_bootstrap.cc:654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.990211 21456 ts_tablet_manager.cc:1434] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20251215 08:58:24.991101 21456 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:24.998174 21456 tablet_bootstrap.cc:654] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:24.998970 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.000516 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.001438 21498 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.004338 21582 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.004926 21582 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.007324 21582 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.011086 21500 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.011706 21500 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.013672 21266 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.014051 21500 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.016018 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.016811 21337 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:25.018363 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.018733 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.018397 21379 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:25.021171 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:25.021884 21381 leader_election.cc:343] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.021981 21415 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:25.023396 21500 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.023912 21500 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.024415 21500 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.025118 21504 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.025635 21504 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.027889 21504 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.030455 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.028753 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:25.030826 21415 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:25.031749 21232 leader_election.cc:343] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.037367 21582 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.037849 21582 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.038265 21582 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.043211 21511 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251215 08:58:25.046484 21307 leader_election.cc:343] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.046603 21504 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:25.051935 21498 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.053674 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:25.059496 21230 leader_election.cc:343] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.060068 21230 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.060248 21467 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.060858 21467 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.061463 21501 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.062856 21467 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.064659 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:25.066043 21232 leader_election.cc:343] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.065918 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.064604 21467 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251215 08:58:25.067409 21230 leader_election.cc:343] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.068032 21230 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.068917 21467 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.070927 21582 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.088626 21457 tablet_bootstrap.cc:492] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.090430 21457 ts_tablet_manager.cc:1403] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.128s user 0.017s sys 0.032s
I20251215 08:58:25.091480 21458 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.091950 21458 ts_tablet_manager.cc:1403] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.118s user 0.027s sys 0.018s
I20251215 08:58:25.093689 21457 raft_consensus.cc:359] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.094455 21457 raft_consensus.cc:385] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.094413 21458 raft_consensus.cc:359] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.094885 21457 raft_consensus.cc:740] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.095819 21457 consensus_queue.cc:260] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.096514 21161 catalog_manager.cc:5654] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:25.095141 21458 raft_consensus.cc:385] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.099076 21458 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.099794 21458 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.101171 21582 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.101312 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:25.101722 21458 ts_tablet_manager.cc:1434] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20251215 08:58:25.102598 21458 tablet_bootstrap.cc:492] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:25.103056 21500 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.104367 21414 raft_consensus.cc:2393] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:25.105490 21500 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.106439 21613 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:25.107311 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.108640 21457 ts_tablet_manager.cc:1434] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20251215 08:58:25.111413 21458 tablet_bootstrap.cc:654] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.112499 21457 tablet_bootstrap.cc:492] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
W20251215 08:58:25.113598 21381 leader_election.cc:343] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.115134 21511 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:25.115659 21511 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.118675 21511 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.120682 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.125221 21458 tablet_bootstrap.cc:492] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.125692 21458 ts_tablet_manager.cc:1403] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.008s
I20251215 08:58:25.127956 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.128106 21458 raft_consensus.cc:359] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.128772 21457 tablet_bootstrap.cc:654] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.128809 21458 raft_consensus.cc:385] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.129215 21458 raft_consensus.cc:740] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.129869 21458 consensus_queue.cc:260] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.130776 21266 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.132747 21456 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.133323 21456 ts_tablet_manager.cc:1403] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.142s user 0.055s sys 0.055s
I20251215 08:58:25.136247 21456 raft_consensus.cc:359] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.136865 21456 raft_consensus.cc:385] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.137125 21456 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.137669 21456 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.139772 21456 ts_tablet_manager.cc:1434] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:58:25.142036 21458 ts_tablet_manager.cc:1434] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20251215 08:58:25.142931 21458 tablet_bootstrap.cc:492] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:25.143039 21457 tablet_bootstrap.cc:492] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.143539 21457 ts_tablet_manager.cc:1403] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.005s
I20251215 08:58:25.144001 21266 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:25.146445 21457 raft_consensus.cc:359] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.147194 21457 raft_consensus.cc:385] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.147508 21457 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
W20251215 08:58:25.147691 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 50909.7us
I20251215 08:58:25.148178 21457 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.149574 21456 tablet_bootstrap.cc:492] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.149889 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:25.150262 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.150702 21337 raft_consensus.cc:2393] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:25.151095 21457 ts_tablet_manager.cc:1434] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20251215 08:58:25.148726 21458 tablet_bootstrap.cc:654] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.152115 21613 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.152630 21613 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.155263 21504 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.155714 21504 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.156679 21456 tablet_bootstrap.cc:654] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.157570 21504 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.158401 21613 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.152112 21457 tablet_bootstrap.cc:492] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.164927 21457 tablet_bootstrap.cc:654] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.166669 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.168670 21264 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:25.186950 21381 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.187805 21613 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.188170 21613 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.188490 21613 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.185380 21498 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.193449 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.193259 21498 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.194047 21339 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:25.195194 21498 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.178472 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.197453 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.198045 21414 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:25.198957 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.199715 21265 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
W20251215 08:58:25.202018 21230 leader_election.cc:343] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.204334 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.206945 21582 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.207293 21582 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.207623 21582 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.213574 21613 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.216395 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.216974 21265 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.218291 21582 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.225512 21265 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:25.226397 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.226751 21381 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.227030 21415 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:25.227828 21573 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.228566 21573 raft_consensus.cc:697] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:25.229462 21573 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.240463 21613 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.240707 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.250759 21163 catalog_manager.cc:5654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:25.255300 21501 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.256052 21582 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.233332 21379 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.263821 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.262172 21501 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.266686 21501 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.261394 21498 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.268067 21498 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.269886 21498 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.262354 21264 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.273620 21499 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.273744 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.274339 21499 raft_consensus.cc:697] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:25.275276 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.275456 21499 consensus_queue.cc:237] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.276427 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.277221 21340 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:25.279222 21615 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251215 08:58:25.282219 21307 leader_election.cc:343] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.283099 21498 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:25.283946 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.285075 21573 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:58:25.291708 21307 consensus_peers.cc:597] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:25.294601 21232 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:25.294765 21266 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:25.295416 21501 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.295951 21501 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.296293 21501 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.306064 21501 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.308169 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:25.309103 21413 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.309213 21501 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.313724 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.314496 21498 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.314915 21498 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.315331 21498 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.322757 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:58:25.325309 21230 leader_election.cc:343] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.329789 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.330816 21415 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:25.348968 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:25.350055 21337 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.354192 21264 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:25.355381 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.356216 21554 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.356676 21554 raft_consensus.cc:697] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:25.357523 21554 consensus_queue.cc:237] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.359091 21457 tablet_bootstrap.cc:492] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.359546 21457 ts_tablet_manager.cc:1403] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.208s user 0.010s sys 0.001s
I20251215 08:58:25.361981 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:25.362757 21415 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.371632 21456 tablet_bootstrap.cc:492] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.372093 21456 ts_tablet_manager.cc:1403] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.223s user 0.056s sys 0.102s
I20251215 08:58:25.372200 21458 tablet_bootstrap.cc:492] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.372632 21458 ts_tablet_manager.cc:1403] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.230s user 0.078s sys 0.118s
I20251215 08:58:25.374496 21456 raft_consensus.cc:359] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.374775 21458 raft_consensus.cc:359] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.375150 21456 raft_consensus.cc:385] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.375331 21458 raft_consensus.cc:385] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.375435 21456 raft_consensus.cc:740] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.375648 21458 raft_consensus.cc:740] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.375536 21503 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:25.376051 21456 consensus_queue.cc:260] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.376242 21458 consensus_queue.cc:260] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.378096 21232 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.352541 21498 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.385581 21456 ts_tablet_manager.cc:1434] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20251215 08:58:25.386575 21456 tablet_bootstrap.cc:492] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.386885 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:58:25.388007 21307 leader_election.cc:343] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.386889 21457 raft_consensus.cc:359] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.389314 21457 raft_consensus.cc:385] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.389645 21457 raft_consensus.cc:740] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.390353 21457 consensus_queue.cc:260] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.393003 21456 tablet_bootstrap.cc:654] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.393898 21498 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.395792 21504 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.395922 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.396632 21265 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.397164 21163 catalog_manager.cc:5654] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:25.398453 21456 tablet_bootstrap.cc:492] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.398882 21456 ts_tablet_manager.cc:1403] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20251215 08:58:25.401222 21456 raft_consensus.cc:359] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.401574 21561 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:25.401875 21456 raft_consensus.cc:385] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.402206 21456 raft_consensus.cc:740] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.402390 21504 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.402837 21456 consensus_queue.cc:260] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.405649 21456 ts_tablet_manager.cc:1434] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251215 08:58:25.406505 21456 tablet_bootstrap.cc:492] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.407774 21413 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:25.411263 21265 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:25.412113 21456 tablet_bootstrap.cc:654] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.413389 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.414386 21629 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.414884 21629 raft_consensus.cc:697] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:25.415726 21629 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.420471 21415 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:25.421639 21232 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:25.422477 21503 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.423038 21503 raft_consensus.cc:697] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:25.424041 21503 consensus_queue.cc:237] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.426764 21458 ts_tablet_manager.cc:1434] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.054s user 0.006s sys 0.001s
I20251215 08:58:25.430032 21458 tablet_bootstrap.cc:492] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:25.435113 21456 tablet_bootstrap.cc:492] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.435583 21456 ts_tablet_manager.cc:1403] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.008s
I20251215 08:58:25.438058 21456 raft_consensus.cc:359] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.438716 21456 raft_consensus.cc:385] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.438978 21456 raft_consensus.cc:740] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.439589 21456 consensus_queue.cc:260] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.440119 21457 ts_tablet_manager.cc:1434] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.080s user 0.008s sys 0.000s
I20251215 08:58:25.440959 21457 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.439249 21337 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:25.445726 21456 ts_tablet_manager.cc:1434] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20251215 08:58:25.446870 21456 tablet_bootstrap.cc:492] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.454353 21163 catalog_manager.cc:5654] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:25.455958 21458 tablet_bootstrap.cc:654] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.465345 21161 catalog_manager.cc:5654] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:25.466853 21161 catalog_manager.cc:5654] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:25.471626 21457 tablet_bootstrap.cc:654] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.474689 21456 tablet_bootstrap.cc:654] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.479712 21573 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:25.481343 21457 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.481735 21457 ts_tablet_manager.cc:1403] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.001s
I20251215 08:58:25.483072 21456 tablet_bootstrap.cc:492] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.483496 21456 ts_tablet_manager.cc:1403] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.005s
I20251215 08:58:25.484452 21457 raft_consensus.cc:359] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.485348 21457 raft_consensus.cc:385] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.485685 21457 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.485970 21456 raft_consensus.cc:359] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.486680 21456 raft_consensus.cc:385] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.486975 21456 raft_consensus.cc:740] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.486459 21457 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.487915 21456 consensus_queue.cc:260] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.489145 21457 ts_tablet_manager.cc:1434] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251215 08:58:25.490188 21456 ts_tablet_manager.cc:1434] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251215 08:58:25.490260 21457 tablet_bootstrap.cc:492] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.491137 21456 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.497620 21456 tablet_bootstrap.cc:654] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.498142 21457 tablet_bootstrap.cc:654] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:25.506765 21381 consensus_peers.cc:597] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Couldn't send request to peer beefe15185ac42ef9a20251007f39a56. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:25.511600 21456 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.512056 21456 ts_tablet_manager.cc:1403] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20251215 08:58:25.513958 21458 tablet_bootstrap.cc:492] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.514387 21458 ts_tablet_manager.cc:1403] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.002s
I20251215 08:58:25.514557 21456 raft_consensus.cc:359] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.515203 21456 raft_consensus.cc:385] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.515471 21456 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.516487 21456 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.516634 21458 raft_consensus.cc:359] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.517468 21458 raft_consensus.cc:385] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.518813 21458 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.518905 21456 ts_tablet_manager.cc:1434] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:25.519934 21456 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.519567 21458 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.517551 21457 tablet_bootstrap.cc:492] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.522045 21458 ts_tablet_manager.cc:1434] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251215 08:58:25.522382 21457 ts_tablet_manager.cc:1403] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.004s
I20251215 08:58:25.525270 21457 raft_consensus.cc:359] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.525810 21456 tablet_bootstrap.cc:654] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.526639 21458 tablet_bootstrap.cc:492] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:25.525839 21457 raft_consensus.cc:385] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.529224 21457 raft_consensus.cc:740] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.530007 21457 consensus_queue.cc:260] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.532577 21458 tablet_bootstrap.cc:654] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.533880 21573 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.534330 21573 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.537343 21457 ts_tablet_manager.cc:1434] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20251215 08:58:25.538304 21573 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.538302 21457 tablet_bootstrap.cc:492] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.539338 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.539551 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.541239 21381 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.542037 21573 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.542471 21629 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.542972 21629 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.545225 21503 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.545856 21503 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.547140 21629 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.548171 21503 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.548492 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.549320 21413 raft_consensus.cc:2393] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:25.549487 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.550048 21415 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:25.550643 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.552047 21232 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.553323 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.553503 21501 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.553857 21501 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.553939 21337 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:25.554208 21501 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.552618 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.557847 21573 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.558303 21629 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.558324 21573 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.564270 21573 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.566447 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:25.567708 21381 leader_election.cc:343] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.567728 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.568742 21337 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:25.570204 21379 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.571393 21573 raft_consensus.cc:2804] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.571756 21573 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.571882 21456 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.572105 21573 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.572382 21456 ts_tablet_manager.cc:1403] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.007s
I20251215 08:58:25.573534 21501 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.574872 21456 raft_consensus.cc:359] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.575531 21456 raft_consensus.cc:385] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.575786 21456 raft_consensus.cc:740] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.576683 21629 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.577111 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:25.578859 21337 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.586411 21457 tablet_bootstrap.cc:654] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.577283 21501 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.576371 21456 consensus_queue.cc:260] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.577827 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:25.585129 21337 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:25.588502 21413 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.589064 21230 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.589838 21456 ts_tablet_manager.cc:1434] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20251215 08:58:25.589924 21501 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.590701 21456 tablet_bootstrap.cc:492] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.591058 21501 raft_consensus.cc:697] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:25.591915 21501 consensus_queue.cc:237] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.594154 21573 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.599699 21458 tablet_bootstrap.cc:492] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.600584 21458 ts_tablet_manager.cc:1403] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.074s user 0.028s sys 0.022s
I20251215 08:58:25.599609 21413 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:25.601039 21456 tablet_bootstrap.cc:654] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.610883 21458 raft_consensus.cc:359] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.611692 21458 raft_consensus.cc:385] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.611994 21458 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.612600 21458 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.618572 21458 ts_tablet_manager.cc:1434] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20251215 08:58:25.619133 21573 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.619123 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.619509 21458 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
W20251215 08:58:25.621788 21381 leader_election.cc:343] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.620574 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:25.624142 21337 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.628429 21457 tablet_bootstrap.cc:492] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.629223 21457 ts_tablet_manager.cc:1403] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.091s user 0.023s sys 0.000s
I20251215 08:58:25.630864 21337 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:25.632362 21379 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.632455 21457 raft_consensus.cc:359] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.633517 21457 raft_consensus.cc:385] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.633533 21573 raft_consensus.cc:2804] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.634104 21457 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.634518 21458 tablet_bootstrap.cc:654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.634981 21573 raft_consensus.cc:697] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:25.634999 21457 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.639417 21617 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:25.640563 21573 consensus_queue.cc:237] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.644274 21457 ts_tablet_manager.cc:1434] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20251215 08:58:25.645246 21457 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.645635 21162 catalog_manager.cc:5654] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:25.651286 21457 tablet_bootstrap.cc:654] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.656006 21617 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.656543 21617 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.658828 21617 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.659665 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.660322 21265 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:25.660562 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.661484 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.661878 21456 tablet_bootstrap.cc:492] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.662324 21456 ts_tablet_manager.cc:1403] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.007s
I20251215 08:58:25.662324 21617 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.662825 21617 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.663184 21617 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.663628 21458 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.663972 21457 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.664062 21458 ts_tablet_manager.cc:1403] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.001s
I20251215 08:58:25.664625 21457 ts_tablet_manager.cc:1403] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
W20251215 08:58:25.662281 21379 leader_election.cc:343] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.666513 21458 raft_consensus.cc:359] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.667150 21458 raft_consensus.cc:385] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.667454 21458 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.667287 21457 raft_consensus.cc:359] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.668224 21457 raft_consensus.cc:385] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.668097 21458 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.668535 21457 raft_consensus.cc:740] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.669297 21457 consensus_queue.cc:260] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.671658 21457 ts_tablet_manager.cc:1434] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:25.672545 21457 tablet_bootstrap.cc:492] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.672631 21458 ts_tablet_manager.cc:1434] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20251215 08:58:25.672461 21162 catalog_manager.cc:5654] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:25.665025 21456 raft_consensus.cc:359] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.673710 21456 raft_consensus.cc:385] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.673997 21456 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.674285 21458 tablet_bootstrap.cc:492] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:25.674609 21456 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.672981 21617 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.678905 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:25.679478 21337 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.680485 21457 tablet_bootstrap.cc:654] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.684680 21613 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.685871 21456 ts_tablet_manager.cc:1434] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20251215 08:58:25.686794 21456 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.689826 21613 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.691764 21613 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.693220 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.693722 21457 tablet_bootstrap.cc:492] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.693835 21264 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.694235 21457 ts_tablet_manager.cc:1403] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.002s
I20251215 08:58:25.696236 21456 tablet_bootstrap.cc:654] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.693238 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.699838 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.700641 21501 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.701114 21501 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.702891 21381 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.703151 21511 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:25.703018 21501 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.704636 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.708065 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.710490 21232 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.711410 21615 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.712030 21501 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.714910 21264 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:25.715881 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:25.716147 21337 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:25.716753 21573 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.716980 21456 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.696981 21457 raft_consensus.cc:359] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.717478 21456 ts_tablet_manager.cc:1403] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.004s
I20251215 08:58:25.717482 21573 raft_consensus.cc:697] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:25.717999 21457 raft_consensus.cc:385] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.718341 21457 raft_consensus.cc:740] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.718935 21573 consensus_queue.cc:237] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.720415 21456 raft_consensus.cc:359] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.721071 21456 raft_consensus.cc:385] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.721349 21456 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.722039 21456 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.724570 21456 ts_tablet_manager.cc:1434] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251215 08:58:25.725816 21456 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.719007 21457 consensus_queue.cc:260] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.735332 21617 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:25.738229 21457 ts_tablet_manager.cc:1434] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20251215 08:58:25.740674 21458 tablet_bootstrap.cc:654] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.740693 21456 tablet_bootstrap.cc:654] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.751097 21457 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.752245 21162 catalog_manager.cc:5654] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:25.769752 21457 tablet_bootstrap.cc:654] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.778187 21458 tablet_bootstrap.cc:492] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.778609 21457 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.778662 21456 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.778641 21458 ts_tablet_manager.cc:1403] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.105s user 0.006s sys 0.009s
I20251215 08:58:25.779218 21456 ts_tablet_manager.cc:1403] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.014s
I20251215 08:58:25.780984 21457 ts_tablet_manager.cc:1403] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.001s
I20251215 08:58:25.781450 21458 raft_consensus.cc:359] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.781668 21456 raft_consensus.cc:359] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.782167 21458 raft_consensus.cc:385] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.782562 21456 raft_consensus.cc:385] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.782608 21458 raft_consensus.cc:740] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.783085 21456 raft_consensus.cc:740] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.783349 21457 raft_consensus.cc:359] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.784135 21457 raft_consensus.cc:385] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.783392 21458 consensus_queue.cc:260] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.783756 21456 consensus_queue.cc:260] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.784417 21457 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.785068 21457 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.786274 21456 ts_tablet_manager.cc:1434] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251215 08:58:25.787052 21457 ts_tablet_manager.cc:1434] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:58:25.787120 21456 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:25.787897 21457 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.788985 21629 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.789450 21629 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.790241 21458 ts_tablet_manager.cc:1434] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20251215 08:58:25.791110 21458 tablet_bootstrap.cc:492] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:25.791354 21629 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.793387 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.793944 21456 tablet_bootstrap.cc:654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:25.795397 21307 leader_election.cc:343] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.798893 21458 tablet_bootstrap.cc:654] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.801818 21630 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.802173 21503 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.802484 21630 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.802793 21503 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.793495 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.810293 21264 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:25.812132 21503 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.813282 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:25.813890 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.815608 21413 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.816495 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:25.817893 21232 leader_election.cc:343] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.816521 21630 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.820441 21415 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:25.820492 21230 leader_election.cc:343] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.821146 21230 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.821661 21457 tablet_bootstrap.cc:654] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:25.822803 21501 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:25.826396 21630 consensus_queue.cc:1048] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:25.828989 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.829617 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.830246 21414 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:25.832140 21629 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.832494 21629 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.832824 21629 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:25.836608 21307 leader_election.cc:343] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.837472 21499 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:25.837641 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.837922 21499 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.839900 21499 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.846939 21457 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:25.847566 21457 ts_tablet_manager.cc:1403] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.005s
I20251215 08:58:25.847105 21629 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.849035 21498 raft_consensus.cc:2804] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.849521 21498 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.849934 21498 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.854218 21458 tablet_bootstrap.cc:492] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:25.853996 21457 raft_consensus.cc:359] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.854671 21457 raft_consensus.cc:385] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.855034 21457 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.855705 21457 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.854643 21458 ts_tablet_manager.cc:1403] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.009s
I20251215 08:58:25.868163 21457 ts_tablet_manager.cc:1434] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.020s user 0.003s sys 0.005s
I20251215 08:58:25.869781 21457 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:25.870420 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:25.870728 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:25.871052 21412 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
W20251215 08:58:25.877367 21307 leader_election.cc:343] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.877928 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:25.878839 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:25.879714 21499 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:25.880061 21499 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:25.880311 21499 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:25.880786 21307 leader_election.cc:343] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.884193 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.884775 21265 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.887224 21499 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.888993 21499 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.874275 21629 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.889298 21498 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.891008 21498 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.899506 21456 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:25.947961 21456 ts_tablet_manager.cc:1403] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.161s user 0.040s sys 0.052s
I20251215 08:58:25.901742 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:25.953730 21411 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.954398 21458 raft_consensus.cc:359] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.955070 21458 raft_consensus.cc:385] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.955340 21458 raft_consensus.cc:740] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.956148 21458 consensus_queue.cc:260] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.900686 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.957522 21628 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:58:25.934252 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:25.945708 21265 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:25.960238 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:25.965560 21414 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.969296 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:25.969378 21628 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.971544 21628 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:25.972342 21628 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:25.972832 21628 raft_consensus.cc:697] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:25.974073 21458 ts_tablet_manager.cc:1434] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.112s user 0.005s sys 0.002s
I20251215 08:58:25.973779 21628 consensus_queue.cc:237] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.977356 21456 raft_consensus.cc:359] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:25.978003 21456 raft_consensus.cc:385] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:25.978297 21456 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:25.979074 21457 tablet_bootstrap.cc:654] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:25.975531 21307 leader_election.cc:343] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.975812 21458 tablet_bootstrap.cc:492] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:25.930253 21629 consensus_queue.cc:1048] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20251215 08:58:25.958853 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:25.979019 21456 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:25.989212 21307 leader_election.cc:343] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:25.990420 21412 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:25.993990 21414 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:25.998189 21456 ts_tablet_manager.cc:1434] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.050s user 0.001s sys 0.006s
I20251215 08:58:25.998909 21456 tablet_bootstrap.cc:492] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:26.003533 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:26.004025 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.009217 21503 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.009668 21503 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.009060 21501 consensus_queue.cc:1048] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.012871 21554 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
W20251215 08:58:26.041675 21307 leader_election.cc:343] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.042117 21503 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.029593 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:26.043082 21629 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.048405 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:26.034396 21554 raft_consensus.cc:697] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:58:26.056918 21307 leader_election.cc:343] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.057422 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:58:26.058430 21230 leader_election.cc:343] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.061621 21411 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:26.066041 21263 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:26.067363 21573 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.068429 21456 tablet_bootstrap.cc:654] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.067977 21573 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.069880 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:26.071079 21629 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:26.071329 21499 raft_consensus.cc:2804] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:26.072073 21499 raft_consensus.cc:697] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:26.072567 21573 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:26.074573 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.075169 21412 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:26.076496 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:26.077087 21266 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:26.072883 21499 consensus_queue.cc:237] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.078912 21161 catalog_manager.cc:5654] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:26.099063 21458 tablet_bootstrap.cc:654] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.103396 21498 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.104547 21630 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.104811 21498 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.103950 21582 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20251215 08:58:26.112318 21498 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.113191 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:26.114804 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.103494 21381 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:26.116577 21617 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:26.116931 21617 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:26.117249 21617 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:26.117321 21307 leader_election.cc:343] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.119055 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:26.119410 21504 consensus_queue.cc:1048] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.120544 21504 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.121682 21504 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:26.125566 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:26.122035 21504 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:58:26.136778 21379 leader_election.cc:343] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.107939 21554 consensus_queue.cc:237] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.133988 21504 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:26.161011 21307 leader_election.cc:343] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.161592 21307 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:26.127696 21628 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20251215 08:58:26.170645 21457 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:26.171232 21457 ts_tablet_manager.cc:1403] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.302s user 0.021s sys 0.017s
I20251215 08:58:26.167474 21582 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:26.210482 21617 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.224493 21458 tablet_bootstrap.cc:492] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:26.225044 21458 ts_tablet_manager.cc:1403] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.249s user 0.012s sys 0.011s
I20251215 08:58:26.228358 21456 tablet_bootstrap.cc:492] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:26.228967 21456 ts_tablet_manager.cc:1403] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.230s user 0.007s sys 0.016s
I20251215 08:58:26.242774 21266 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:26.244138 21456 raft_consensus.cc:359] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.244810 21456 raft_consensus.cc:385] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.249513 21613 consensus_queue.cc:1048] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.250862 21613 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.254973 21456 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:26.257002 21457 raft_consensus.cc:359] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.257721 21457 raft_consensus.cc:385] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.257965 21457 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
W20251215 08:58:26.258371 21674 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.053s user 0.003s sys 0.000s
I20251215 08:58:26.258632 21457 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.261921 21617 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:26.265630 21456 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.266525 21457 ts_tablet_manager.cc:1434] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.095s user 0.007s sys 0.001s
I20251215 08:58:26.271771 21458 raft_consensus.cc:359] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.272569 21458 raft_consensus.cc:385] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.272790 21458 raft_consensus.cc:740] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
W20251215 08:58:26.273613 21673 log.cc:927] Time spent T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.065s user 0.002s sys 0.000s
I20251215 08:58:26.273531 21458 consensus_queue.cc:260] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.221514 21504 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.284093 21458 ts_tablet_manager.cc:1434] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.059s user 0.005s sys 0.000s
I20251215 08:58:26.285409 21458 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:26.287856 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:26.289161 21554 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.289594 21554 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.291347 21554 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:58:26.275898 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 56678.7us
I20251215 08:58:26.295200 21456 ts_tablet_manager.cc:1434] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.065s user 0.007s sys 0.000s
I20251215 08:58:26.296633 21456 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:26.300153 21457 tablet_bootstrap.cc:492] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:26.300055 21681 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251215 08:58:26.295173 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:26.320009 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:26.321249 21669 log.cc:927] Time spent T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.062s user 0.002s sys 0.000s
W20251215 08:58:26.321466 21307 leader_election.cc:343] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.310370 21264 raft_consensus.cc:2393] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:26.350592 21499 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.365166 21504 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.372856 21677 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.363497 21499 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.366814 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:26.383844 21677 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.385206 21617 consensus_queue.cc:1048] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.386938 21499 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:58:26.382346 21379 leader_election.cc:343] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.371632 21573 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20251215 08:58:26.396535 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.397146 21415 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:26.398916 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:26.399690 21498 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:26.400030 21498 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:26.400352 21498 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:26.394651 21261 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:26.388201 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:26.416494 21381 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:26.417390 21617 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251215 08:58:26.421996 21458 tablet_bootstrap.cc:654] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.423014 21677 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.428326 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:26.428953 21261 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:26.430094 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:26.431099 21409 raft_consensus.cc:2393] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:58:26.434453 21230 leader_election.cc:343] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.440438 21456 tablet_bootstrap.cc:654] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.442615 21498 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.518170 21164 catalog_manager.cc:5654] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:26.575132 21499 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.581235 21498 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.581518 21499 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.583351 21499 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.585516 21677 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.586092 21677 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.588140 21677 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.591053 21164 catalog_manager.cc:5654] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:26.598191 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:26.599112 21504 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251215 08:58:26.613723 21458 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:26.614312 21458 ts_tablet_manager.cc:1403] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.329s user 0.027s sys 0.039s
I20251215 08:58:26.617635 21458 raft_consensus.cc:359] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.618386 21458 raft_consensus.cc:385] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.618665 21458 raft_consensus.cc:740] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:26.449389 21307 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:58:26.619505 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 170223us
I20251215 08:58:26.619347 21458 consensus_queue.cc:260] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.619899 21499 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:26.620429 21499 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:26.620846 21499 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:26.621500 21458 ts_tablet_manager.cc:1434] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20251215 08:58:26.622407 21458 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:26.625420 21457 tablet_bootstrap.cc:654] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.627811 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:26.629060 21410 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:26.629238 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.629418 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.634554 21499 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.635317 21615 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.635792 21615 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.636437 21499 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.640426 21615 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:26.640919 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:26.639694 21410 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:26.647102 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:26.648244 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:26.648360 21381 leader_election.cc:343] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.653343 21456 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:26.653781 21456 ts_tablet_manager.cc:1403] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.357s user 0.028s sys 0.052s
I20251215 08:58:26.655413 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:26.655867 21261 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:26.661507 21456 raft_consensus.cc:359] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.662002 21456 raft_consensus.cc:385] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.662204 21456 raft_consensus.cc:740] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:26.662695 21456 consensus_queue.cc:260] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.664582 21456 ts_tablet_manager.cc:1434] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251215 08:58:26.668884 21261 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:26.669831 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:26.670581 21456 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:26.678563 21458 tablet_bootstrap.cc:654] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.679807 21456 tablet_bootstrap.cc:654] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.685698 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:26.687322 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.691766 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:26.693975 21339 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
W20251215 08:58:26.694651 21307 leader_election.cc:343] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.694608 21499 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:26.695546 21499 raft_consensus.cc:697] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:26.696619 21499 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.693487 21457 tablet_bootstrap.cc:492] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:26.702436 21457 ts_tablet_manager.cc:1403] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.402s user 0.025s sys 0.034s
I20251215 08:58:26.705363 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:26.706112 21259 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:26.709359 21307 consensus_peers.cc:597] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:26.708067 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:26.727730 21554 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:26.730468 21230 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:26.730576 21457 raft_consensus.cc:359] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.731565 21457 raft_consensus.cc:385] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.731889 21457 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
W20251215 08:58:26.732602 21230 leader_election.cc:343] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.732955 21681 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:26.740392 21456 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:26.740835 21456 ts_tablet_manager.cc:1403] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.070s user 0.017s sys 0.003s
I20251215 08:58:26.741907 21503 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.742529 21503 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.743643 21457 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.758478 21503 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.759424 21457 ts_tablet_manager.cc:1434] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20251215 08:58:26.760317 21457 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
W20251215 08:58:26.767375 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 55513.4us
W20251215 08:58:26.767910 21307 leader_election.cc:343] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.768330 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:26.769721 21629 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:26.769994 21458 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:26.770480 21458 ts_tablet_manager.cc:1403] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.148s user 0.015s sys 0.030s
W20251215 08:58:26.771199 21307 leader_election.cc:343] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.760923 21259 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:26.774273 21458 raft_consensus.cc:359] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.774904 21458 raft_consensus.cc:385] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.775131 21458 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:26.775756 21458 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:26.766131 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54635.6us
I20251215 08:58:26.777499 21379 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:26.778630 21615 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:26.778975 21615 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:26.779305 21615 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:26.774351 21499 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251215 08:58:26.781164 21458 ts_tablet_manager.cc:1434] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251215 08:58:26.780613 21456 raft_consensus.cc:359] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.782084 21456 raft_consensus.cc:385] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.782094 21458 tablet_bootstrap.cc:492] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:26.782423 21456 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:26.783345 21456 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.786377 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:26.786967 21339 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:26.786863 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:26.789741 21456 ts_tablet_manager.cc:1434] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20251215 08:58:26.790652 21628 consensus_queue.cc:1048] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.795449 21458 tablet_bootstrap.cc:654] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.795105 21615 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:26.798447 21307 consensus_peers.cc:597] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Couldn't send request to peer beefe15185ac42ef9a20251007f39a56. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:26.798560 21615 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:26.799419 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:26.800920 21307 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:58:26.801467 21381 leader_election.cc:343] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.801637 21554 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:26.802083 21554 raft_consensus.cc:697] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:58:26.803218 21232 leader_election.cc:343] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.802928 21554 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.803609 21232 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:26.804769 21503 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:26.806736 21458 tablet_bootstrap.cc:492] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:26.807389 21458 ts_tablet_manager.cc:1403] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.007s
I20251215 08:58:26.808214 21456 tablet_bootstrap.cc:492] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:26.808671 21457 tablet_bootstrap.cc:654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.810639 21458 raft_consensus.cc:359] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.811349 21458 raft_consensus.cc:385] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251215 08:58:26.811609 21232 leader_election.cc:343] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:26.811654 21458 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:26.812676 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:26.813251 21340 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:26.812559 21458 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.820115 21230 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:26.821580 21501 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:26.822042 21501 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:26.822386 21501 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:26.822657 21458 ts_tablet_manager.cc:1434] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20251215 08:58:26.824362 21458 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:26.830267 21458 tablet_bootstrap.cc:654] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:26.830814 21262 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:26.839589 21457 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:26.840056 21457 ts_tablet_manager.cc:1403] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.009s
I20251215 08:58:26.841647 21164 catalog_manager.cc:5654] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 2, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:26.843644 21501 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.851989 21457 raft_consensus.cc:359] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.852739 21457 raft_consensus.cc:385] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:26.852998 21457 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:26.853633 21457 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.855089 21457 ts_tablet_manager.cc:1434] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.015s user 0.004s sys 0.001s
I20251215 08:58:26.855921 21457 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:26.858239 21340 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:26.859432 21379 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:26.861385 21615 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:26.859170 21501 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.861884 21615 raft_consensus.cc:697] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:26.862836 21615 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.869474 21554 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:26.874392 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:26.943871 21458 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:26.944386 21458 ts_tablet_manager.cc:1403] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.120s user 0.019s sys 0.006s
W20251215 08:58:26.944818 21232 leader_election.cc:343] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:26.951076 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 73810.1us
I20251215 08:58:26.961861 21677 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:26.962427 21677 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:26.964730 21677 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:26.993224 21457 tablet_bootstrap.cc:654] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.021476 21501 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.022017 21501 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.024068 21501 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.027653 21554 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.028126 21554 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.025671 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:27.032284 21338 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.032980 21554 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.034691 21458 raft_consensus.cc:359] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.035352 21458 raft_consensus.cc:385] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.035599 21458 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.036201 21458 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.036430 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.038837 21338 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:27.043256 21456 tablet_bootstrap.cc:654] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.044611 21613 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.044713 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.045084 21613 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.045328 21413 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:27.047482 21613 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:27.054838 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:27.059757 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.063714 21458 ts_tablet_manager.cc:1434] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.119s user 0.013s sys 0.013s
I20251215 08:58:27.072227 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.072988 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.074139 21261 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:27.074678 21230 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:27.076265 21677 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:27.076828 21677 raft_consensus.cc:697] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:27.077808 21677 consensus_queue.cc:237] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.088964 21458 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
W20251215 08:58:27.095736 21307 leader_election.cc:343] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.090765 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:27.100018 21232 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:27.101876 21681 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:27.102339 21681 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:27.107900 21681 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.108471 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:27.116310 21458 tablet_bootstrap.cc:654] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:27.117695 21307 leader_election.cc:343] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.118347 21307 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:27.120502 21230 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:27.122795 21161 catalog_manager.cc:5654] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:27.123934 21681 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.106210 21339 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:27.119367 21499 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.128446 21503 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:27.131024 21379 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:27.132156 21694 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.135496 21681 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.140419 21162 catalog_manager.cc:5654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:27.145728 21457 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:27.151161 21457 ts_tablet_manager.cc:1403] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.295s user 0.012s sys 0.015s
W20251215 08:58:27.157155 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:27.162297 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:27.163123 21413 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.170001 21458 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:27.170480 21458 ts_tablet_manager.cc:1403] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.007s
I20251215 08:58:27.172852 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:27.173228 21164 catalog_manager.cc:5654] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:27.174124 21340 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.175383 21458 raft_consensus.cc:359] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.176026 21458 raft_consensus.cc:385] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.176275 21458 raft_consensus.cc:740] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.177994 21413 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:27.179682 21340 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:27.181813 21230 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:27.176844 21458 consensus_queue.cc:260] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.184810 21503 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:27.185282 21503 raft_consensus.cc:697] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:27.186098 21503 consensus_queue.cc:237] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.189469 21694 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.199307 21457 raft_consensus.cc:359] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.200026 21456 tablet_bootstrap.cc:492] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.200521 21456 ts_tablet_manager.cc:1403] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.392s user 0.015s sys 0.028s
I20251215 08:58:27.201272 21458 ts_tablet_manager.cc:1434] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20251215 08:58:27.203488 21456 raft_consensus.cc:359] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.204146 21456 raft_consensus.cc:385] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.204406 21456 raft_consensus.cc:740] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.205037 21456 consensus_queue.cc:260] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.206902 21456 ts_tablet_manager.cc:1434] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251215 08:58:27.207557 21456 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.213078 21456 tablet_bootstrap.cc:654] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.200032 21457 raft_consensus.cc:385] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.216817 21457 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.224435 21503 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.224939 21503 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.226907 21503 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.223428 21458 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:27.238225 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:27.240007 21340 raft_consensus.cc:2393] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:27.233740 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.249106 21232 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:27.250293 21503 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.250690 21412 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251215 08:58:27.250327 21164 catalog_manager.cc:5654] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:27.252580 21554 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:27.266744 21456 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.261763 21503 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.269824 21260 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251215 08:58:27.280988 21456 ts_tablet_manager.cc:1403] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.023s
I20251215 08:58:27.222748 21457 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.267947 21554 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20251215 08:58:27.283707 21456 raft_consensus.cc:359] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.284382 21456 raft_consensus.cc:385] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.284652 21456 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.285244 21456 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.285573 21499 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:27.291359 21456 ts_tablet_manager.cc:1434] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251215 08:58:27.292315 21456 tablet_bootstrap.cc:492] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.298075 21456 tablet_bootstrap.cc:654] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.305133 21695 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:27.308813 21456 tablet_bootstrap.cc:492] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.309257 21456 ts_tablet_manager.cc:1403] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20251215 08:58:27.312608 21456 raft_consensus.cc:359] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.319854 21456 raft_consensus.cc:385] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.320108 21456 raft_consensus.cc:740] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.321581 21456 consensus_queue.cc:260] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.324357 21677 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.324641 21457 ts_tablet_manager.cc:1434] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.173s user 0.008s sys 0.000s
I20251215 08:58:27.324868 21677 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.325690 21457 tablet_bootstrap.cc:492] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:27.326877 21456 ts_tablet_manager.cc:1434] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20251215 08:58:27.326830 21677 leader_election.cc:290] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.327493 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:27.327816 21456 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.328091 21339 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:27.329401 21230 leader_election.cc:304] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:27.329407 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.336103 21677 raft_consensus.cc:2804] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:27.336479 21677 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:27.336807 21677 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.339684 21456 tablet_bootstrap.cc:654] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:27.341856 21232 leader_election.cc:343] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.356509 21458 tablet_bootstrap.cc:654] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.384564 21456 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.385200 21456 ts_tablet_manager.cc:1403] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.058s user 0.019s sys 0.020s
I20251215 08:58:27.393241 21456 raft_consensus.cc:359] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.393692 21456 raft_consensus.cc:385] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.393879 21456 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.394317 21456 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.395076 21629 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.395530 21629 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.397223 21629 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.398198 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.398298 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.398326 21677 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:27.399740 21307 leader_election.cc:343] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:27.403002 21307 leader_election.cc:343] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.403466 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:27.406664 21629 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.408461 21694 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:27.409147 21677 leader_election.cc:290] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.409827 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:27.410321 21339 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.411820 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:58:27.413522 21232 leader_election.cc:343] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.414175 21629 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.414614 21629 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.416229 21456 ts_tablet_manager.cc:1434] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20251215 08:58:27.417066 21457 tablet_bootstrap.cc:654] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.417838 21456 tablet_bootstrap.cc:492] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.416287 21629 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.423349 21458 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:27.423957 21458 ts_tablet_manager.cc:1403] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.202s user 0.020s sys 0.012s
I20251215 08:58:27.425668 21339 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:27.426685 21458 raft_consensus.cc:359] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.427376 21458 raft_consensus.cc:385] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.428074 21458 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.428716 21456 tablet_bootstrap.cc:654] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.428757 21458 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.431123 21458 ts_tablet_manager.cc:1434] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:27.432014 21458 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:27.432103 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.432922 21260 raft_consensus.cc:2393] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:27.434839 21230 leader_election.cc:304] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:27.435114 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.435804 21413 raft_consensus.cc:2393] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:27.436537 21503 raft_consensus.cc:2804] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:27.436988 21503 raft_consensus.cc:697] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:27.437753 21503 consensus_queue.cc:237] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.438455 21458 tablet_bootstrap.cc:654] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.438482 21457 tablet_bootstrap.cc:492] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:27.439110 21457 ts_tablet_manager.cc:1403] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.000s
I20251215 08:58:27.439356 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:27.443663 21457 raft_consensus.cc:359] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.444353 21457 raft_consensus.cc:385] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.444654 21457 raft_consensus.cc:740] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.446147 21613 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:27.445341 21457 consensus_queue.cc:260] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.451301 21629 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.452229 21457 ts_tablet_manager.cc:1434] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20251215 08:58:27.454417 21457 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:27.453544 21456 tablet_bootstrap.cc:492] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.455279 21456 ts_tablet_manager.cc:1403] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.004s
I20251215 08:58:27.459189 21456 raft_consensus.cc:359] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.459832 21629 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.461656 21456 raft_consensus.cc:385] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.462010 21456 raft_consensus.cc:740] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.462729 21456 consensus_queue.cc:260] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.464567 21456 ts_tablet_manager.cc:1434] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:27.465494 21456 tablet_bootstrap.cc:492] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.471007 21458 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:27.471491 21458 ts_tablet_manager.cc:1403] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.007s
I20251215 08:58:27.473977 21458 raft_consensus.cc:359] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.474612 21458 raft_consensus.cc:385] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.474870 21458 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.475468 21458 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.478338 21458 ts_tablet_manager.cc:1434] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251215 08:58:27.479175 21458 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:27.483901 21456 tablet_bootstrap.cc:654] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.487237 21457 tablet_bootstrap.cc:654] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.491312 21458 tablet_bootstrap.cc:654] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.494912 21613 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.495366 21613 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.497184 21613 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:27.498616 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.500234 21161 catalog_manager.cc:5654] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:27.501166 21260 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:27.502246 21381 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:27.503276 21613 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:27.503679 21613 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:27.504033 21613 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.510032 21457 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:27.510649 21457 ts_tablet_manager.cc:1403] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.006s
I20251215 08:58:27.498721 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:27.512774 21339 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:27.516418 21456 tablet_bootstrap.cc:492] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.516863 21456 ts_tablet_manager.cc:1403] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.011s
I20251215 08:58:27.517223 21457 raft_consensus.cc:359] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.517925 21457 raft_consensus.cc:385] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.518175 21457 raft_consensus.cc:740] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.518821 21457 consensus_queue.cc:260] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.519256 21456 raft_consensus.cc:359] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.519982 21456 raft_consensus.cc:385] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.520270 21456 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.520887 21456 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.527207 21456 ts_tablet_manager.cc:1434] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251215 08:58:27.528108 21456 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.530648 21457 ts_tablet_manager.cc:1434] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20251215 08:58:27.531522 21457 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:27.539080 21458 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:27.539652 21458 ts_tablet_manager.cc:1403] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.001s
I20251215 08:58:27.540149 21456 tablet_bootstrap.cc:654] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.542145 21458 raft_consensus.cc:359] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.542924 21458 raft_consensus.cc:385] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.543249 21458 raft_consensus.cc:740] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.543905 21458 consensus_queue.cc:260] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.546798 21458 ts_tablet_manager.cc:1434] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:27.548880 21613 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.561712 21457 tablet_bootstrap.cc:654] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.563948 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:27.567528 21339 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.564705 21681 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:27.571938 21458 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:27.574559 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:27.575176 21260 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.579901 21456 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.580627 21456 ts_tablet_manager.cc:1403] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.007s
I20251215 08:58:27.590448 21339 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:27.592387 21260 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:27.592335 21681 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:27.596486 21260 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.601894 21613 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:27.593995 21381 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:27.607756 21458 tablet_bootstrap.cc:654] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.610167 21456 raft_consensus.cc:359] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.610915 21456 raft_consensus.cc:385] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.611147 21456 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.611712 21456 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.613339 21413 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.615875 21613 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:27.620019 21613 raft_consensus.cc:697] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:27.620901 21457 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:27.620792 21613 consensus_queue.cc:237] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.621330 21457 ts_tablet_manager.cc:1403] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.004s
I20251215 08:58:27.624615 21456 ts_tablet_manager.cc:1434] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.044s user 0.006s sys 0.001s
I20251215 08:58:27.639743 21456 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.650591 21457 raft_consensus.cc:359] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.651389 21457 raft_consensus.cc:385] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.651718 21457 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.652794 21457 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.671873 21554 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.680774 21457 ts_tablet_manager.cc:1434] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20251215 08:58:27.687590 21499 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.688189 21499 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.677101 21554 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.690024 21499 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.691461 21554 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.691793 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:27.697057 21307 leader_election.cc:343] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.700188 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.701695 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:27.703696 21307 leader_election.cc:343] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.710899 21457 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
W20251215 08:58:27.733631 21232 consensus_peers.cc:597] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251215 08:58:27.724328 21307 leader_election.cc:343] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.715842 21695 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.741402 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:58:27.742118 21729 log.cc:927] Time spent T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20251215 08:58:27.741957 21695 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.742290 21554 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.724282 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:27.745637 21307 leader_election.cc:343] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.746043 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:27.747538 21554 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.748788 21456 tablet_bootstrap.cc:654] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.717926 21629 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:58:27.717633 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 46564.3us
I20251215 08:58:27.761613 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.762248 21259 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:27.717056 21503 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.041s
I20251215 08:58:27.767158 21307 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:27.764789 21163 catalog_manager.cc:5654] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:27.754508 21629 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.778683 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.777145 21736 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:27.791256 21736 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:27.791632 21736 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.791873 21413 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:27.792408 21629 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.780107 21695 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.781193 21457 tablet_bootstrap.cc:654] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.797523 21458 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:27.797964 21458 ts_tablet_manager.cc:1403] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.226s user 0.021s sys 0.006s
I20251215 08:58:27.799994 21458 raft_consensus.cc:359] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.800464 21458 raft_consensus.cc:385] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.800670 21458 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.807942 21736 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.808071 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.809862 21736 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.810735 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:27.817368 21307 leader_election.cc:343] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.818081 21695 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.818701 21695 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.819531 21458 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.831513 21458 ts_tablet_manager.cc:1434] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20251215 08:58:27.831334 21695 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.832309 21458 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:27.826048 21260 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:27.836495 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:27.840258 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:27.840790 21413 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.840947 21263 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.845263 21307 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:27.852401 21413 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:27.845613 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:27.848569 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.848874 21573 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:27.855409 21573 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.857704 21499 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251215 08:58:27.858389 21307 leader_election.cc:343] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.860062 21307 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:27.863147 21695 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
W20251215 08:58:27.863248 21307 leader_election.cc:343] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.863639 21695 raft_consensus.cc:697] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:27.864166 21456 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.863664 21307 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:27.864666 21456 ts_tablet_manager.cc:1403] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.225s user 0.030s sys 0.026s
I20251215 08:58:27.858059 21499 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:27.864490 21695 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.866555 21499 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.868696 21573 leader_election.cc:290] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:27.869810 21414 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.865137 21554 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.870834 21263 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:27.873236 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:27.873952 21339 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:27.879968 21503 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:27.876705 21499 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.882982 21735 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251215 08:58:27.884898 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:27.885501 21263 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:27.885890 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:27.887416 21381 leader_election.cc:304] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:27.888805 21613 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:27.889933 21457 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:27.890410 21457 ts_tablet_manager.cc:1403] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.209s user 0.015s sys 0.010s
I20251215 08:58:27.889648 21456 raft_consensus.cc:359] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.892304 21456 raft_consensus.cc:385] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.892585 21456 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.892774 21457 raft_consensus.cc:359] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.893478 21457 raft_consensus.cc:385] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.893237 21456 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.893724 21457 raft_consensus.cc:740] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.894361 21457 consensus_queue.cc:260] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.895653 21456 ts_tablet_manager.cc:1434] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20251215 08:58:27.896508 21499 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:27.896653 21457 ts_tablet_manager.cc:1434] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251215 08:58:27.900738 21457 tablet_bootstrap.cc:492] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:27.913522 21457 tablet_bootstrap.cc:654] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.896696 21456 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.897325 21458 tablet_bootstrap.cc:654] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.898056 21263 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:27.899408 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:27.930461 21456 tablet_bootstrap.cc:654] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.931316 21307 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:58:27.932066 21307 leader_election.cc:343] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:27.934217 21499 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:27.934440 21458 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:27.934873 21458 ts_tablet_manager.cc:1403] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.103s user 0.013s sys 0.001s
I20251215 08:58:27.934996 21499 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:27.935775 21499 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.937155 21458 raft_consensus.cc:359] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.937819 21458 raft_consensus.cc:385] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.938068 21458 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.938802 21458 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.940327 21457 tablet_bootstrap.cc:492] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:27.944162 21457 ts_tablet_manager.cc:1403] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.000s
I20251215 08:58:27.945500 21458 ts_tablet_manager.cc:1434] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20251215 08:58:27.946337 21458 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:27.946409 21457 raft_consensus.cc:359] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.947070 21457 raft_consensus.cc:385] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.947289 21457 raft_consensus.cc:740] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.947213 21163 catalog_manager.cc:5654] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:27.947940 21457 consensus_queue.cc:260] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.953747 21458 tablet_bootstrap.cc:654] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.957307 21456 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:27.957882 21456 ts_tablet_manager.cc:1403] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.000s
I20251215 08:58:27.961185 21457 ts_tablet_manager.cc:1434] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20251215 08:58:27.962121 21457 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:27.970537 21457 tablet_bootstrap.cc:654] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.965572 21456 raft_consensus.cc:359] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.974418 21456 raft_consensus.cc:385] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:27.974694 21456 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:27.975250 21456 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:27.976073 21161 catalog_manager.cc:5654] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:27.977939 21456 ts_tablet_manager.cc:1434] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20251215 08:58:27.981794 21456 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:27.996222 21458 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:27.996733 21458 ts_tablet_manager.cc:1403] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.023s
I20251215 08:58:27.999394 21456 tablet_bootstrap.cc:654] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:27.999434 21458 raft_consensus.cc:359] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.000453 21458 raft_consensus.cc:385] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.000711 21458 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.001317 21458 consensus_queue.cc:260] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.003485 21458 ts_tablet_manager.cc:1434] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251215 08:58:28.003935 21457 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:28.004307 21457 ts_tablet_manager.cc:1403] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.009s
I20251215 08:58:28.004899 21458 tablet_bootstrap.cc:492] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:28.006673 21457 raft_consensus.cc:359] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.007194 21457 raft_consensus.cc:385] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.007395 21457 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.007918 21457 consensus_queue.cc:260] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.010025 21457 ts_tablet_manager.cc:1434] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251215 08:58:28.010612 21554 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.011065 21554 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.012023 21457 tablet_bootstrap.cc:492] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:28.015523 21458 tablet_bootstrap.cc:654] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.021509 21456 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.021858 21456 ts_tablet_manager.cc:1403] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.004s
I20251215 08:58:28.023681 21456 raft_consensus.cc:359] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.024292 21456 raft_consensus.cc:385] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.024556 21456 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.026083 21456 consensus_queue.cc:260] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.028249 21456 ts_tablet_manager.cc:1434] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251215 08:58:28.012929 21554 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.029230 21456 tablet_bootstrap.cc:492] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:28.029398 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.029949 21263 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:28.033401 21457 tablet_bootstrap.cc:654] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.036450 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.040444 21456 tablet_bootstrap.cc:654] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.041953 21458 tablet_bootstrap.cc:492] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:28.042775 21458 ts_tablet_manager.cc:1403] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.010s
I20251215 08:58:28.045125 21458 raft_consensus.cc:359] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.046842 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.047605 21554 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.047958 21554 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.048291 21554 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.048432 21457 tablet_bootstrap.cc:492] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:28.048906 21457 ts_tablet_manager.cc:1403] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.004s
I20251215 08:58:28.051409 21457 raft_consensus.cc:359] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.052228 21457 raft_consensus.cc:385] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.052496 21457 raft_consensus.cc:740] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
W20251215 08:58:28.053369 21307 leader_election.cc:343] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.054316 21554 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.054912 21458 raft_consensus.cc:385] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.055209 21458 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.056000 21456 tablet_bootstrap.cc:492] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.056169 21554 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.055922 21458 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.056465 21456 ts_tablet_manager.cc:1403] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.009s
I20251215 08:58:28.058220 21458 ts_tablet_manager.cc:1434] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20251215 08:58:28.058903 21456 raft_consensus.cc:359] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.059348 21458 tablet_bootstrap.cc:492] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:28.059567 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:28.060111 21263 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.060205 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:28.053112 21457 consensus_queue.cc:260] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.062958 21457 ts_tablet_manager.cc:1434] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20251215 08:58:28.063921 21457 tablet_bootstrap.cc:492] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:28.065094 21456 raft_consensus.cc:385] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.065400 21456 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.066092 21456 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.067231 21263 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
W20251215 08:58:28.070796 21307 leader_election.cc:343] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.067675 21456 ts_tablet_manager.cc:1434] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251215 08:58:28.071475 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:28.071924 21458 tablet_bootstrap.cc:654] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.072346 21456 tablet_bootstrap.cc:492] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:28.074678 21457 tablet_bootstrap.cc:654] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.074581 21629 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:28.075160 21629 raft_consensus.cc:697] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:28.075960 21629 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.078969 21736 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.079403 21736 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.081418 21736 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.081892 21456 tablet_bootstrap.cc:654] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.091427 21457 tablet_bootstrap.cc:492] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:28.091921 21457 ts_tablet_manager.cc:1403] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.004s
I20251215 08:58:28.094647 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.094861 21457 raft_consensus.cc:359] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.095562 21457 raft_consensus.cc:385] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.095906 21457 raft_consensus.cc:740] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.096673 21457 consensus_queue.cc:260] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.098719 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.099054 21457 ts_tablet_manager.cc:1434] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
W20251215 08:58:28.101150 21307 leader_election.cc:343] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.102928 21458 tablet_bootstrap.cc:492] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:28.104924 21458 ts_tablet_manager.cc:1403] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.002s
I20251215 08:58:28.107419 21458 raft_consensus.cc:359] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:28.108999 21307 leader_election.cc:343] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.109160 21458 raft_consensus.cc:385] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.109527 21458 raft_consensus.cc:740] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.109570 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:28.110437 21629 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:28.110710 21458 consensus_queue.cc:260] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.112330 21458 ts_tablet_manager.cc:1434] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251215 08:58:28.113200 21458 tablet_bootstrap.cc:492] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:28.117470 21458 tablet_bootstrap.cc:654] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.120643 21573 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:28.121022 21615 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.121454 21615 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.123585 21615 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.125444 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.126317 21340 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:28.127750 21379 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:28.130106 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:28.131843 21381 leader_election.cc:343] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.133060 21615 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.133140 21613 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.133562 21613 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.133865 21613 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.139957 21613 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.143263 21615 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.146065 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.148826 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.150467 21379 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:28.150671 21457 tablet_bootstrap.cc:492] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:28.151304 21573 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:28.157639 21457 tablet_bootstrap.cc:654] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.152690 21615 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.161576 21456 tablet_bootstrap.cc:492] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.161971 21456 ts_tablet_manager.cc:1403] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.090s user 0.025s sys 0.023s
I20251215 08:58:28.163512 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:28.164563 21456 raft_consensus.cc:359] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:28.164746 21381 leader_election.cc:343] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.165218 21456 raft_consensus.cc:385] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.165474 21456 raft_consensus.cc:740] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.166028 21456 consensus_queue.cc:260] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.169263 21456 ts_tablet_manager.cc:1434] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251215 08:58:28.170130 21456 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:28.167447 21615 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:28.175571 21456 tablet_bootstrap.cc:654] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.177412 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:28.178031 21340 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.184618 21458 tablet_bootstrap.cc:492] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:28.185134 21458 ts_tablet_manager.cc:1403] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.027s
I20251215 08:58:28.193781 21573 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.194362 21573 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.196295 21573 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.197693 21456 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.197751 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.198247 21456 ts_tablet_manager.cc:1403] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.005s
I20251215 08:58:28.198431 21339 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:28.200107 21457 tablet_bootstrap.cc:492] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:28.200706 21457 ts_tablet_manager.cc:1403] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.002s
I20251215 08:58:28.200763 21379 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:28.202391 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.203037 21259 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:28.203501 21456 raft_consensus.cc:359] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.204101 21456 raft_consensus.cc:385] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.204396 21456 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.204075 21573 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.205294 21573 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.205770 21573 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.204126 21458 raft_consensus.cc:359] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.205467 21456 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.206236 21458 raft_consensus.cc:385] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.206661 21458 raft_consensus.cc:740] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.207429 21458 consensus_queue.cc:260] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.207759 21456 ts_tablet_manager.cc:1434] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20251215 08:58:28.211890 21456 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:28.213812 21458 ts_tablet_manager.cc:1434] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20251215 08:58:28.214980 21458 tablet_bootstrap.cc:492] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:28.209107 21613 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:28.209333 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:28.210291 21457 raft_consensus.cc:359] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.216625 21457 raft_consensus.cc:385] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.216987 21457 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.217779 21456 tablet_bootstrap.cc:654] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.218848 21457 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.221078 21457 ts_tablet_manager.cc:1434] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.020s user 0.003s sys 0.004s
I20251215 08:58:28.221910 21457 tablet_bootstrap.cc:492] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:28.242058 21340 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:28.243871 21458 tablet_bootstrap.cc:654] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.248879 21379 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:28.249711 21456 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.249660 21615 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:28.250120 21456 ts_tablet_manager.cc:1403] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.020s
I20251215 08:58:28.250718 21457 tablet_bootstrap.cc:654] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.251500 21615 raft_consensus.cc:697] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:28.257071 21615 consensus_queue.cc:237] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.266916 21573 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.271384 21456 raft_consensus.cc:359] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.271853 21456 raft_consensus.cc:385] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.272094 21456 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.274096 21456 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.274902 21163 catalog_manager.cc:5654] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:28.279116 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:28.279699 21337 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.279832 21456 ts_tablet_manager.cc:1434] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.029s user 0.004s sys 0.001s
I20251215 08:58:28.280697 21456 tablet_bootstrap.cc:492] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:28.287881 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:28.277251 21573 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.288713 21259 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.301116 21409 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.310848 21458 tablet_bootstrap.cc:492] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:28.312569 21458 ts_tablet_manager.cc:1403] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.098s user 0.016s sys 0.017s
I20251215 08:58:28.313632 21457 tablet_bootstrap.cc:492] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:28.314073 21457 ts_tablet_manager.cc:1403] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.004s
I20251215 08:58:28.316242 21457 raft_consensus.cc:359] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.316017 21161 catalog_manager.cc:5654] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:28.316798 21457 raft_consensus.cc:385] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.318711 21457 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.319646 21259 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:28.318341 21767 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.320304 21767 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.324545 21381 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.325711 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.326318 21615 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:28.326799 21615 raft_consensus.cc:697] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:28.327387 21503 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:58:28.327433 21379 leader_election.cc:343] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.327824 21503 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.329293 21337 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:28.329345 21767 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.319422 21457 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.331525 21456 tablet_bootstrap.cc:654] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.332139 21615 consensus_queue.cc:237] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.335098 21458 raft_consensus.cc:359] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.335779 21458 raft_consensus.cc:385] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.336024 21458 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.337107 21458 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.337425 21457 ts_tablet_manager.cc:1434] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20251215 08:58:28.337603 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.338325 21414 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:28.338726 21457 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:28.340437 21232 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.341346 21734 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.341696 21734 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.342006 21734 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.342914 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.343531 21337 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:28.345652 21458 ts_tablet_manager.cc:1434] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20251215 08:58:28.346487 21458 tablet_bootstrap.cc:492] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:28.348842 21695 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.349319 21695 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.351523 21695 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.351680 21458 tablet_bootstrap.cc:654] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.354424 21456 tablet_bootstrap.cc:492] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.357406 21456 ts_tablet_manager.cc:1403] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.001s
I20251215 08:58:28.359931 21456 raft_consensus.cc:359] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.360567 21456 raft_consensus.cc:385] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.354534 21737 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.361688 21737 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.363773 21737 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.356252 21554 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.364683 21554 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.366675 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.367364 21458 tablet_bootstrap.cc:492] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:28.367517 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.367802 21458 ts_tablet_manager.cc:1403] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.002s
I20251215 08:58:28.368113 21338 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
W20251215 08:58:28.368702 21307 consensus_peers.cc:597] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251215 08:58:28.369230 21307 leader_election.cc:343] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.369689 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.369926 21554 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.360817 21456 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.370888 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.371474 21261 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:28.372822 21458 raft_consensus.cc:359] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.373523 21458 raft_consensus.cc:385] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.329401 21503 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.374132 21458 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.374221 21456 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.374818 21458 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.376152 21456 ts_tablet_manager.cc:1434] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20251215 08:58:28.376878 21456 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:28.377161 21458 ts_tablet_manager.cc:1434] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251215 08:58:28.378034 21458 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:28.378746 21767 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.379186 21767 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.381832 21767 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.373378 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.382645 21414 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:28.381767 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.383613 21261 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:28.383651 21232 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.352670 21457 tablet_bootstrap.cc:654] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.384718 21381 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.384975 21503 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.382005 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.385355 21503 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.385643 21338 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:28.386409 21767 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.386821 21767 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.387125 21767 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.387388 21456 tablet_bootstrap.cc:654] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.385658 21503 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.391665 21457 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:28.392045 21457 ts_tablet_manager.cc:1403] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.002s
I20251215 08:58:28.326948 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.393743 21458 tablet_bootstrap.cc:654] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.394281 21457 raft_consensus.cc:359] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.394976 21457 raft_consensus.cc:385] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.395265 21457 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.396783 21503 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.398769 21737 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.399046 21503 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.399216 21737 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.399806 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:28.400507 21413 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.400753 21737 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.401603 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.402158 21410 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:28.402973 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.403059 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.403872 21737 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:28.404219 21737 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.404579 21737 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:28.408658 21413 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:28.409998 21232 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.413148 21681 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:28.415407 21457 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.418818 21457 ts_tablet_manager.cc:1434] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20251215 08:58:28.422334 21457 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:28.422242 21695 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.422791 21695 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.425093 21695 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.404336 21259 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:28.426544 21737 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.428262 21381 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:28.429292 21615 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.429661 21615 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.430083 21615 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.431357 21737 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.432737 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:28.441351 21414 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:28.438172 21615 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.445189 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:28.445848 21337 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.449532 21615 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.377671 21734 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.449688 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:28.475118 21338 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.461390 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.476243 21457 tablet_bootstrap.cc:654] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.455744 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:28.477510 21259 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.478579 21456 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.480154 21456 ts_tablet_manager.cc:1403] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.002s
I20251215 08:58:28.476601 21339 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:28.481369 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:28.481923 21340 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:28.484684 21337 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
W20251215 08:58:28.493626 21307 leader_election.cc:343] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.494043 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:28.482805 21161 catalog_manager.cc:5654] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:28.496176 21307 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.503302 21456 raft_consensus.cc:359] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.506402 21456 raft_consensus.cc:385] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.506747 21456 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.509001 21456 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.510521 21338 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:28.511062 21456 ts_tablet_manager.cc:1434] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.031s user 0.004s sys 0.001s
I20251215 08:58:28.511591 21379 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:28.511962 21456 tablet_bootstrap.cc:492] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:28.512414 21615 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:28.512881 21615 raft_consensus.cc:697] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:28.513684 21615 consensus_queue.cc:237] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:28.518993 21307 leader_election.cc:343] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.498142 21695 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:28.499608 21629 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:28.534950 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.528307 21629 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:28.530786 21734 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.535984 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:28.534093 21414 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:58:28.529155 21767 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.542927 21629 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.540940 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:28.540558 21681 raft_consensus.cc:697] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:28.584182 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:28.575564 21338 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.588901 21458 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:28.589965 21458 ts_tablet_manager.cc:1403] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.212s user 0.035s sys 0.048s
I20251215 08:58:28.584829 21409 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.592412 21737 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:28.593814 21458 raft_consensus.cc:359] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.596280 21735 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.593530 21681 consensus_queue.cc:237] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.598075 21767 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.601461 21737 raft_consensus.cc:697] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:28.606241 21737 consensus_queue.cc:237] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.606999 21259 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:28.601272 21458 raft_consensus.cc:385] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.609676 21458 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.610630 21456 tablet_bootstrap.cc:654] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.602418 21735 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.602408 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.603291 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:28.610213 21458 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.620018 21629 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.622913 21629 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.625200 21261 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.625521 21338 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:28.627902 21458 ts_tablet_manager.cc:1434] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.038s user 0.005s sys 0.001s
I20251215 08:58:28.631392 21340 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:58:28.634593 21629 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:28.640096 21458 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:28.639405 21779 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251215 08:58:28.639377 21629 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251215 08:58:28.642484 21735 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.645283 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:28.656752 21573 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:58:28.656785 21307 leader_election.cc:343] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.657510 21573 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.660933 21573 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:28.645414 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:58:28.664232 21307 leader_election.cc:343] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.664810 21307 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:28.666265 21736 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:28.672106 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.675051 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.668100 21695 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251215 08:58:28.668547 21340 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.680027 21161 catalog_manager.cc:5654] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:28.681190 21262 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:28.687757 21457 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:28.688367 21457 ts_tablet_manager.cc:1403] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.266s user 0.022s sys 0.005s
I20251215 08:58:28.688911 21458 tablet_bootstrap.cc:654] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:28.696347 21457 raft_consensus.cc:359] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.698130 21457 raft_consensus.cc:385] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:28.697252 21379 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:28.700373 21457 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:28.703096 21457 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.702720 21163 catalog_manager.cc:5654] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:28.702379 21767 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.719784 21695 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:28.720456 21695 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.722935 21695 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:28.725363 21767 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:28.726971 21409 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:28.729880 21261 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:28.730041 21232 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:58:28.733623 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 66680.6us
I20251215 08:58:28.735036 21340 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:28.736366 21379 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:28.740895 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:28.741518 21261 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.752795 21573 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:28.759227 21573 raft_consensus.cc:697] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:28.761229 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:28.742970 21681 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:28.762233 21457 ts_tablet_manager.cc:1434] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.074s user 0.008s sys 0.000s
I20251215 08:58:28.762676 21681 raft_consensus.cc:697] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:28.763552 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:28.762224 21340 raft_consensus.cc:2393] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:28.759989 21573 consensus_queue.cc:237] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:28.810686 21230 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:28.811875 21780 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:28.851402 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:28.857604 21457 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
W20251215 08:58:28.858693 21307 leader_election.cc:343] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.862884 21681 consensus_queue.cc:237] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:28.869084 21307 leader_election.cc:343] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:28.982908 21456 tablet_bootstrap.cc:492] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:28.983600 21456 ts_tablet_manager.cc:1403] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.472s user 0.030s sys 0.038s
I20251215 08:58:28.994603 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:29.029071 21794 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:29.029572 21794 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.033846 21794 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:29.024582 21797 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20251215 08:58:29.043632 21307 leader_election.cc:343] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:29.044039 21307 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:29.050531 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:29.051012 21340 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:29.046490 21796 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:29.042966 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:29.065685 21456 raft_consensus.cc:359] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.066419 21456 raft_consensus.cc:385] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.066679 21456 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.073457 21456 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.061848 21798 consensus_queue.cc:1048] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20251215 08:58:29.077994 21261 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:29.082093 21230 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:29.083324 21503 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:29.083680 21503 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:29.083992 21503 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:29.087060 21458 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:29.088516 21456 ts_tablet_manager.cc:1434] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.105s user 0.003s sys 0.004s
I20251215 08:58:29.089308 21458 ts_tablet_manager.cc:1403] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.449s user 0.020s sys 0.004s
I20251215 08:58:29.090019 21613 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:29.099189 21503 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.101835 21503 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:29.102558 21780 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:29.092650 21737 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20251215 08:58:29.112644 21457 tablet_bootstrap.cc:654] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:29.115994 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 52172us
I20251215 08:58:29.117319 21458 raft_consensus.cc:359] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.099025 21798 consensus_queue.cc:1048] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20251215 08:58:29.118208 21802 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:29.118978 21458 raft_consensus.cc:385] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.120569 21458 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.123241 21802 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.121136 21458 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.126788 21802 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:29.128515 21456 tablet_bootstrap.cc:492] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:29.129436 21790 consensus_queue.cc:1048] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251215 08:58:29.127699 21458 ts_tablet_manager.cc:1434] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20251215 08:58:29.145731 21458 tablet_bootstrap.cc:492] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:29.150662 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:58:29.157701 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 67947.5us
W20251215 08:58:29.159737 21232 leader_election.cc:343] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:29.162853 21458 tablet_bootstrap.cc:654] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
W20251215 08:58:29.163581 21232 leader_election.cc:343] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:29.164167 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:29.164783 21334 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:29.170277 21789 log.cc:927] Time spent T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.204s user 0.004s sys 0.005s
I20251215 08:58:29.176112 21458 tablet_bootstrap.cc:492] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:29.176669 21458 ts_tablet_manager.cc:1403] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.000s
I20251215 08:58:29.179811 21458 raft_consensus.cc:359] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.180894 21458 raft_consensus.cc:385] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.181216 21458 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.185241 21458 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.188362 21458 ts_tablet_manager.cc:1434] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20251215 08:58:29.189579 21458 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:29.190968 21334 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:29.201400 21790 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:29.202342 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:29.201979 21790 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.218819 21777 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251215 08:58:29.226835 21230 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:29.225700 21792 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251215 08:58:29.231618 21737 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:29.232156 21737 raft_consensus.cc:697] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:29.235199 21307 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:29.204909 21798 consensus_queue.cc:1048] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20251215 08:58:29.236318 21457 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:29.237578 21457 ts_tablet_manager.cc:1403] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.380s user 0.012s sys 0.014s
I20251215 08:58:29.244113 21456 tablet_bootstrap.cc:654] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:29.233815 21737 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:29.275324 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:29.260689 21790 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:29.283181 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:29.283946 21337 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:29.284799 21799 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:29.285451 21799 raft_consensus.cc:697] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:29.286428 21799 consensus_queue.cc:237] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.265923 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:29.289587 21263 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:29.308422 21161 catalog_manager.cc:5654] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 2, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:29.314272 21161 catalog_manager.cc:5654] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:29.328661 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:29.332587 21379 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:29.336956 21457 raft_consensus.cc:359] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.337822 21457 raft_consensus.cc:385] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.338151 21457 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.336292 21790 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:29.340644 21790 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:29.341089 21790 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:29.342913 21258 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:29.338943 21457 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.364265 21456 tablet_bootstrap.cc:492] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:29.364668 21456 ts_tablet_manager.cc:1403] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.236s user 0.010s sys 0.007s
I20251215 08:58:29.377923 21407 raft_consensus.cc:1275] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20251215 08:58:29.380570 21810 log.cc:927] Time spent T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20251215 08:58:29.381848 21503 consensus_queue.cc:1048] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:29.384862 21457 ts_tablet_manager.cc:1434] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.145s user 0.007s sys 0.000s
I20251215 08:58:29.393157 21457 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:29.405226 21457 tablet_bootstrap.cc:654] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:29.418704 21458 tablet_bootstrap.cc:654] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:29.485636 21456 raft_consensus.cc:359] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.486675 21456 raft_consensus.cc:385] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.486986 21456 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.498790 21456 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:29.520227 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 265340us
W20251215 08:58:29.520903 21307 leader_election.cc:343] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:29.525676 21337 raft_consensus.cc:1275] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251215 08:58:29.537288 21456 ts_tablet_manager.cc:1434] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.172s user 0.009s sys 0.005s
I20251215 08:58:29.538268 21456 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:29.524389 21737 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20251215 08:58:29.538419 21790 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.540889 21790 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:29.548329 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:29.556317 21785 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:29.556706 21785 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:29.557006 21785 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:29.557248 21615 consensus_queue.cc:1048] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251215 08:58:29.639927 21785 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.619678 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:29.645663 21333 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:29.651844 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:58:29.652460 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 93738.4us
I20251215 08:58:29.667739 21457 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:29.668200 21457 ts_tablet_manager.cc:1403] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.275s user 0.019s sys 0.007s
I20251215 08:58:29.652479 21261 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:29.669852 21458 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:29.670248 21458 ts_tablet_manager.cc:1403] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.481s user 0.019s sys 0.005s
I20251215 08:58:29.668210 21780 consensus_queue.cc:1048] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251215 08:58:29.685976 21456 tablet_bootstrap.cc:654] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:29.693312 21785 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:29.693648 21458 raft_consensus.cc:359] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.694294 21458 raft_consensus.cc:385] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.694526 21458 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.695044 21458 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.709321 21457 raft_consensus.cc:359] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.710022 21457 raft_consensus.cc:385] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.710294 21457 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.678817 21777 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20251215 08:58:29.718690 21499 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:29.718757 21457 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:29.741801 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 74108.7us
I20251215 08:58:29.719245 21458 ts_tablet_manager.cc:1434] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20251215 08:58:29.774550 21458 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:29.783102 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:29.783610 21834 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:29.783748 21457 ts_tablet_manager.cc:1434] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.115s user 0.011s sys 0.018s
I20251215 08:58:29.784062 21834 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.785125 21457 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:29.785970 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:29.785866 21499 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.791857 21499 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:29.723201 21261 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
W20251215 08:58:29.794901 21307 leader_election.cc:343] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:29.789534 21265 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:29.793870 21792 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:29.807706 21792 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.809623 21333 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:29.816409 21613 consensus_queue.cc:1048] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251215 08:58:29.816167 21381 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:29.823252 21615 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:29.834458 21615 raft_consensus.cc:697] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:29.835284 21615 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.834002 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:29.830621 21792 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:29.843770 21834 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:29.851805 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:29.852481 21266 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:29.852741 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:29.855947 21338 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:29.858809 21456 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:29.858629 21499 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:29.859313 21456 ts_tablet_manager.cc:1403] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.321s user 0.036s sys 0.010s
I20251215 08:58:29.860508 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:29.846884 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:29.862643 21379 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:58:29.882433 21307 leader_election.cc:343] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:29.889840 21456 raft_consensus.cc:359] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.892493 21456 raft_consensus.cc:385] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:29.878825 21798 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:29.892807 21456 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:29.878527 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:29.893460 21456 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.890837 21265 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:29.894680 21798 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:29.919329 21456 ts_tablet_manager.cc:1434] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.060s user 0.007s sys 0.002s
I20251215 08:58:29.925532 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:29.926303 21499 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:29.920929 21798 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:29.945478 21499 raft_consensus.cc:697] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:29.912132 21805 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20251215 08:58:29.948029 21499 consensus_queue.cc:237] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:29.954308 21458 tablet_bootstrap.cc:654] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:29.959733 21336 raft_consensus.cc:2393] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:29.951237 21785 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:29.980906 21456 tablet_bootstrap.cc:492] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:29.988149 21457 tablet_bootstrap.cc:654] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.004838 21164 catalog_manager.cc:5654] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:30.006721 21379 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:29.987617 21785 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.014930 21613 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.075685 21785 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.088635 21798 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.151414 21798 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:30.154795 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 60802.7us
W20251215 08:58:30.155728 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 174460us
I20251215 08:58:30.159432 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:30.166144 21457 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:30.166607 21457 ts_tablet_manager.cc:1403] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.382s user 0.023s sys 0.012s
I20251215 08:58:30.176409 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:30.184783 21458 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:30.185256 21458 ts_tablet_manager.cc:1403] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.411s user 0.043s sys 0.056s
I20251215 08:58:30.169047 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:30.188859 21341 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.192577 21259 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.192870 21458 raft_consensus.cc:359] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.195091 21458 raft_consensus.cc:385] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.195382 21458 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.196002 21458 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.249899 21458 ts_tablet_manager.cc:1434] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.064s user 0.003s sys 0.003s
I20251215 08:58:30.250752 21458 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
W20251215 08:58:30.252188 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 85513.3us
W20251215 08:58:30.260592 21307 leader_election.cc:343] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.263053 21613 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251215 08:58:30.274173 21307 leader_election.cc:343] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.274559 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:58:30.287106 21307 consensus_peers.cc:597] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:30.281548 21629 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:30.292099 21457 raft_consensus.cc:359] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.293179 21457 raft_consensus.cc:385] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.293445 21457 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.295085 21457 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.299744 21785 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.300278 21785 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.302160 21785 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.304085 21341 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:30.309556 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:30.314831 21307 leader_election.cc:343] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.318197 21835 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:30.323108 21456 tablet_bootstrap.cc:654] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.332518 21457 ts_tablet_manager.cc:1434] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.166s user 0.004s sys 0.002s
I20251215 08:58:30.333963 21457 tablet_bootstrap.cc:492] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:30.334827 21379 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:30.343101 21458 tablet_bootstrap.cc:654] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.338501 21615 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:30.340507 21848 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251215 08:58:30.349553 21615 raft_consensus.cc:697] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:30.352825 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:30.371456 21499 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.371884 21499 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.373603 21499 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.374971 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:30.375737 21263 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:30.381989 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:30.370314 21259 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:30.390100 21861 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:30.394130 21861 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:30.394487 21861 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.376686 21615 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.398759 21164 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36424 (request call id 96) took 1190 ms. Trace:
W20251215 08:58:30.401141 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:30.401491 21163 catalog_manager.cc:5654] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:30.399171 21164 rpcz_store.cc:276] 1215 08:58:29.208441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:29.208572 (+ 131us) service_pool.cc:225] Handling call
1215 08:58:30.398679 (+1190107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:58:30.017250 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:58:30.018040 (+ 790us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:58:30.018084 (+ 44us) write_op.cc:435] Acquired schema lock
1215 08:58:30.018118 (+ 34us) tablet.cc:660] Decoding operations
1215 08:58:30.149452 (+131334us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:58:30.151621 (+ 2169us) write_op.cc:641] Partition lock acquired for write op
1215 08:58:30.156698 (+ 5077us) tablet.cc:683] Acquiring locks for 1 operations
1215 08:58:30.168499 (+ 11801us) tablet.cc:699] Row locks acquired
1215 08:58:30.169169 (+ 670us) write_op.cc:260] PREPARE: finished
1215 08:58:30.178036 (+ 8867us) write_op.cc:270] Start()
1215 08:58:30.182894 (+ 4858us) write_op.cc:276] Timestamp: P: 1765789110176920 usec, L: 0
1215 08:58:30.187176 (+ 4282us) op_driver.cc:348] REPLICATION: starting
1215 08:58:30.271093 (+ 83917us) log.cc:844] Serialized 540 byte log entry
1215 08:58:30.273158 (+ 2065us) op_driver.cc:464] REPLICATION: finished
1215 08:58:30.274986 (+ 1828us) write_op.cc:301] APPLY: starting
1215 08:58:30.275071 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
1215 08:58:30.275176 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
1215 08:58:30.275264 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:58:30.276821 (+ 1557us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:58:30.288123 (+ 11302us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1215 08:58:30.291882 (+ 3759us) write_op.cc:312] APPLY: finished
1215 08:58:30.297372 (+ 5490us) log.cc:844] Serialized 22 byte log entry
1215 08:58:30.300087 (+ 2715us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:58:30.300201 (+ 114us) write_op.cc:454] Released schema lock
1215 08:58:30.300238 (+ 37us) write_op.cc:341] FINISH: Updating metrics
1215 08:58:30.300544 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:58:30.300569 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":41405,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":221,"apply.run_cpu_time_us":1683,"apply.run_wall_time_us":26937,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":24128,"prepare.run_wall_time_us":255172,"replication_time_us":85846}]]}
I20251215 08:58:30.447197 21161 catalog_manager.cc:5654] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:30.452852 21790 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.469314 21790 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.471268 21790 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:30.481732 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:30.482059 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:30.483368 21457 tablet_bootstrap.cc:654] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.488092 21861 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.491693 21861 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.492226 21456 tablet_bootstrap.cc:492] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:30.493453 21456 ts_tablet_manager.cc:1403] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.512s user 0.044s sys 0.061s
I20251215 08:58:30.494673 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:30.495255 21258 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.519578 21799 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:30.519989 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:30.523676 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:58:30.528191 21307 leader_election.cc:343] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.527339 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:30.529686 21379 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:30.531312 21307 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:58:30.533452 21307 leader_election.cc:343] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.533771 21307 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:30.532938 21736 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:30.539853 21823 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251215 08:58:30.543449 21307 leader_election.cc:343] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.536751 21834 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251215 08:58:30.546401 21307 leader_election.cc:343] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.537992 21858 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251215 08:58:30.548815 21163 catalog_manager.cc:5654] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:30.561255 21258 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
W20251215 08:58:30.564184 21307 consensus_peers.cc:597] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Couldn't send request to peer 195a56ac72a84018880bfb3bf6e6ef1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:30.567795 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:30.569245 21799 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:30.569715 21799 raft_consensus.cc:697] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:30.570482 21799 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.571601 21458 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:30.572053 21458 ts_tablet_manager.cc:1403] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.322s user 0.009s sys 0.025s
I20251215 08:58:30.572326 21456 raft_consensus.cc:359] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.580348 21456 raft_consensus.cc:385] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.580701 21456 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.581372 21456 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.586620 21456 ts_tablet_manager.cc:1434] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.093s user 0.006s sys 0.001s
I20251215 08:58:30.591111 21456 tablet_bootstrap.cc:492] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:30.595820 21458 raft_consensus.cc:359] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.596561 21458 raft_consensus.cc:385] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.596889 21458 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.600214 21457 tablet_bootstrap.cc:492] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:30.600737 21457 ts_tablet_manager.cc:1403] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.267s user 0.013s sys 0.013s
I20251215 08:58:30.602380 21161 catalog_manager.cc:5654] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:30.603379 21457 raft_consensus.cc:359] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.604207 21457 raft_consensus.cc:385] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.604496 21457 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.605170 21457 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.597827 21458 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.606197 21161 catalog_manager.cc:5654] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:30.608189 21457 ts_tablet_manager.cc:1434] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:30.609119 21457 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:30.614622 21457 tablet_bootstrap.cc:654] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.610316 21458 ts_tablet_manager.cc:1434] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.038s user 0.007s sys 0.001s
I20251215 08:58:30.617584 21458 tablet_bootstrap.cc:492] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:30.632023 21456 tablet_bootstrap.cc:654] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.636844 21457 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:30.637404 21457 ts_tablet_manager.cc:1403] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.007s
I20251215 08:58:30.637840 21458 tablet_bootstrap.cc:654] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.640350 21457 raft_consensus.cc:359] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.641196 21457 raft_consensus.cc:385] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.641453 21457 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.642043 21457 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.643986 21457 ts_tablet_manager.cc:1434] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20251215 08:58:30.644866 21457 tablet_bootstrap.cc:492] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:30.656998 21457 tablet_bootstrap.cc:654] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.673595 21457 tablet_bootstrap.cc:492] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:30.674055 21457 ts_tablet_manager.cc:1403] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.005s
I20251215 08:58:30.678231 21456 tablet_bootstrap.cc:492] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:30.678687 21456 ts_tablet_manager.cc:1403] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.088s user 0.016s sys 0.003s
I20251215 08:58:30.679018 21457 raft_consensus.cc:359] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.679680 21457 raft_consensus.cc:385] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.679909 21457 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.680505 21457 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.681069 21456 raft_consensus.cc:359] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.681747 21456 raft_consensus.cc:385] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.682019 21456 raft_consensus.cc:740] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.680759 21163 catalog_manager.cc:5654] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:30.682962 21457 ts_tablet_manager.cc:1434] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251215 08:58:30.682631 21456 consensus_queue.cc:260] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.683652 21834 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.684083 21834 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.686080 21834 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:30.686738 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:30.687495 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:30.689442 21457 tablet_bootstrap.cc:492] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:30.690076 21458 tablet_bootstrap.cc:492] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:30.690502 21458 ts_tablet_manager.cc:1403] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.073s user 0.017s sys 0.000s
I20251215 08:58:30.691787 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:30.693994 21834 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:30.696627 21458 raft_consensus.cc:359] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.697366 21458 raft_consensus.cc:385] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.697825 21458 raft_consensus.cc:740] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.699301 21456 ts_tablet_manager.cc:1434] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20251215 08:58:30.699414 21458 consensus_queue.cc:260] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.700222 21456 tablet_bootstrap.cc:492] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4: Bootstrap starting.
I20251215 08:58:30.701840 21458 ts_tablet_manager.cc:1434] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20251215 08:58:30.708945 21458 tablet_bootstrap.cc:492] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:30.714735 21458 tablet_bootstrap.cc:654] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.715586 21457 tablet_bootstrap.cc:654] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.719664 21456 tablet_bootstrap.cc:654] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.730799 21456 tablet_bootstrap.cc:492] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4: No bootstrap required, opened a new log
I20251215 08:58:30.731271 21456 ts_tablet_manager.cc:1403] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.005s
I20251215 08:58:30.734182 21456 raft_consensus.cc:359] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.734915 21456 raft_consensus.cc:385] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.735188 21456 raft_consensus.cc:740] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.735874 21456 consensus_queue.cc:260] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.743240 21456 ts_tablet_manager.cc:1434] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251215 08:58:30.746719 21458 tablet_bootstrap.cc:492] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:30.747231 21458 ts_tablet_manager.cc:1403] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.002s
I20251215 08:58:30.752038 21408 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.773129 21799 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.773656 21799 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.776306 21799 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.779429 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:30.781864 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:30.783465 21307 leader_election.cc:343] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:30.784454 21307 leader_election.cc:343] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.785410 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:30.786407 21799 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:30.793181 21458 raft_consensus.cc:359] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.801196 21458 raft_consensus.cc:385] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.801438 21458 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.802084 21458 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.803807 21458 ts_tablet_manager.cc:1434] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.056s user 0.003s sys 0.000s
I20251215 08:58:30.810138 21458 tablet_bootstrap.cc:492] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:30.812420 21457 tablet_bootstrap.cc:492] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:30.812922 21457 ts_tablet_manager.cc:1403] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.124s user 0.013s sys 0.008s
I20251215 08:58:30.815686 21458 tablet_bootstrap.cc:654] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.815553 21457 raft_consensus.cc:359] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.816226 21457 raft_consensus.cc:385] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.816495 21457 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.826604 21615 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.827059 21615 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.830199 21613 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:30.832638 21780 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.833240 21780 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.845497 21858 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.846026 21858 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.846379 21615 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:30.848184 21858 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:30.848747 21458 tablet_bootstrap.cc:492] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:30.849444 21458 ts_tablet_manager.cc:1403] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.002s
I20251215 08:58:30.849563 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:30.849931 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:30.850258 21334 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:30.850648 21794 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.851131 21794 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.851909 21379 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:30.852773 21858 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:30.852571 21457 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.853156 21858 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:30.853528 21858 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.855672 21457 ts_tablet_manager.cc:1434] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20251215 08:58:30.855635 21458 raft_consensus.cc:359] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.856351 21458 raft_consensus.cc:385] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.856626 21458 raft_consensus.cc:740] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.858153 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:30.858767 21259 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:30.858676 21458 consensus_queue.cc:260] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.861001 21458 ts_tablet_manager.cc:1434] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251215 08:58:30.862291 21794 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.863137 21457 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:30.863217 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:30.865654 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:30.870483 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:30.871649 21615 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:30.874190 21858 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.875007 21412 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.875337 21407 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 1: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:30.877360 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:30.877961 21336 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:30.878513 21457 tablet_bootstrap.cc:654] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.880319 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:30.880506 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:30.880878 21336 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
W20251215 08:58:30.881902 21232 leader_election.cc:343] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:30.882413 21780 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.885169 21230 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:30.886740 21458 tablet_bootstrap.cc:492] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:30.886807 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:30.887188 21794 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:30.887996 21257 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.889119 21858 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:30.890749 21336 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:30.892016 21379 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:30.896242 21257 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:30.896705 21794 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:30.898705 21458 tablet_bootstrap.cc:654] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:30.899480 21457 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:30.899955 21457 ts_tablet_manager.cc:1403] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.008s
I20251215 08:58:30.897099 21794 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:30.911005 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:30.911654 21336 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:30.912070 21458 tablet_bootstrap.cc:492] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:30.912585 21458 ts_tablet_manager.cc:1403] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.005s
I20251215 08:58:30.913916 21230 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:30.914920 21780 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:30.915257 21780 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:30.915591 21780 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.918860 21615 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:30.919395 21615 raft_consensus.cc:697] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:30.920212 21615 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.922147 21780 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.922459 21457 raft_consensus.cc:359] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.923270 21457 raft_consensus.cc:385] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.923574 21457 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.921424 21458 raft_consensus.cc:359] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.924858 21458 raft_consensus.cc:385] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:30.925212 21458 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:30.926101 21458 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.927011 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:30.927572 21336 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.925769 21794 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.932269 21503 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:30.932706 21503 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.934228 21503 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.935930 21458 ts_tablet_manager.cc:1434] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.023s user 0.002s sys 0.003s
I20251215 08:58:30.937080 21458 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: Bootstrap starting.
I20251215 08:58:30.924131 21457 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:30.939837 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:30.940415 21334 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:30.940629 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:30.946391 21794 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.946988 21341 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 0.
I20251215 08:58:30.950476 21407 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.956158 21457 ts_tablet_manager.cc:1434] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.056s user 0.010s sys 0.005s
I20251215 08:58:30.959710 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:30.960929 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:30.962917 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:30.963857 21794 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:30.964232 21794 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:30.964567 21794 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:30.973543 21780 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:30.975180 21765 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:30.981460 21457 tablet_bootstrap.cc:492] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:30.992710 21792 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:31.011644 21736 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:31.012151 21736 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.013922 21336 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:31.014108 21736 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:31.015605 21794 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.017453 21230 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:58:31.017639 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 55402.4us
I20251215 08:58:31.018303 21503 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:31.018739 21503 raft_consensus.cc:697] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:31.005059 21334 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:58:31.021973 21457 tablet_bootstrap.cc:654] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:31.019510 21503 consensus_queue.cc:237] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.006088 21792 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.027896 21230 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:31.034830 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:31.041345 21307 leader_election.cc:343] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:31.070325 21413 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.031754 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:31.087679 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:31.065672 21850 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election won for term 2
W20251215 08:58:31.112797 21232 leader_election.cc:343] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
W20251215 08:58:31.115430 21232 leader_election.cc:343] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:31.108438 21850 raft_consensus.cc:697] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:31.099718 21257 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 0.
I20251215 08:58:31.131044 21794 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:31.139318 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:31.145397 21850 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.146418 21823 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:31.146775 21823 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:31.147060 21823 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.147653 21458 tablet_bootstrap.cc:654] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:31.156787 21457 tablet_bootstrap.cc:492] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:31.157449 21457 ts_tablet_manager.cc:1403] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.176s user 0.016s sys 0.000s
I20251215 08:58:31.142692 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:31.161160 21792 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:31.163180 21613 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:31.163666 21613 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.171582 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:31.166723 21232 leader_election.cc:343] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:31.175207 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:31.176777 21613 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:31.185580 21266 raft_consensus.cc:2393] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:31.187101 21736 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:31.193295 21381 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:31.196943 21613 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:31.202072 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:31.202751 21266 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:31.204208 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:31.198180 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:31.214871 21457 raft_consensus.cc:359] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.215569 21457 raft_consensus.cc:385] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:31.215801 21457 raft_consensus.cc:740] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:31.218128 21457 consensus_queue.cc:260] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.224849 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:31.225539 21333 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:31.208055 21615 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:31.229525 21615 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:31.229871 21615 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.211208 21341 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.232682 21823 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.237766 21823 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:31.249734 21457 ts_tablet_manager.cc:1434] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.092s user 0.007s sys 0.000s
I20251215 08:58:31.251056 21457 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
W20251215 08:58:31.251453 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 51145.7us
I20251215 08:58:31.276947 21799 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:31.291252 21161 catalog_manager.cc:5654] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:31.277520 21615 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.308629 21457 tablet_bootstrap.cc:654] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:31.311915 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:31.318491 21260 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.318866 21341 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:31.320034 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:31.320608 21266 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.361639 21458 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: No bootstrap required, opened a new log
I20251215 08:58:31.365144 21458 ts_tablet_manager.cc:1403] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: Time spent bootstrapping tablet: real 0.428s user 0.035s sys 0.044s
I20251215 08:58:31.369266 21615 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:31.370777 21260 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:31.370793 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:31.375305 21266 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:31.376468 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:31.377528 21858 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:31.378029 21858 raft_consensus.cc:697] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:31.378916 21858 consensus_queue.cc:237] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:31.410708 21307 leader_election.cc:343] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Illegal state: must be running to vote when last-logged opid is not known
I20251215 08:58:31.413858 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:31.414906 21850 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:31.415356 21850 raft_consensus.cc:697] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:31.416146 21850 consensus_queue.cc:237] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.425861 21458 raft_consensus.cc:359] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.433100 21458 raft_consensus.cc:385] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:31.433385 21458 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Initialized, Role: FOLLOWER
I20251215 08:58:31.434092 21458 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.443789 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:31.444442 21334 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.462342 21613 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:31.463008 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:31.465320 21823 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:31.465778 21823 raft_consensus.cc:697] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:31.466588 21823 consensus_queue.cc:237] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.462913 21613 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.472522 21458 ts_tablet_manager.cc:1434] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: Time spent starting tablet: real 0.107s user 0.007s sys 0.000s
I20251215 08:58:31.476490 21613 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:31.478703 21212 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251215 08:58:31.494719 21457 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:31.495358 21457 ts_tablet_manager.cc:1403] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.245s user 0.017s sys 0.012s
W20251215 08:58:31.475706 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 50186.5us
I20251215 08:58:31.514971 21412 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.576429 21334 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:31.527858 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:31.588491 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:31.589293 21337 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 1: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:31.591118 21457 raft_consensus.cc:359] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.591993 21457 raft_consensus.cc:385] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:31.592322 21457 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:31.600579 21259 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:31.599069 21457 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.611660 21379 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:31.621924 21457 ts_tablet_manager.cc:1434] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.126s user 0.003s sys 0.013s
I20251215 08:58:31.622859 21457 tablet_bootstrap.cc:492] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:31.622712 21615 consensus_queue.cc:1059] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:31.623406 21615 consensus_queue.cc:1243] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer beefe15185ac42ef9a20251007f39a56 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251215 08:58:31.623930 21615 raft_consensus.cc:3055] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:31.624217 21615 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:31.624963 21615 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.634405 21615 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:31.642055 21336 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:31.645920 21613 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251215 08:58:31.646538 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 47348.2us
I20251215 08:58:31.649659 21503 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:31.661942 21457 tablet_bootstrap.cc:654] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:31.674563 21780 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:31.685995 21332 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251215 08:58:31.685621 21162 catalog_manager.cc:5654] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:31.687107 21162 catalog_manager.cc:5654] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 2, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:31.694332 21615 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:31.694774 21615 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.697120 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:31.697671 21258 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:31.698598 21381 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:31.699517 21613 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 08:58:31.699877 21613 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:31.700176 21613 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.703346 21885 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:31.727447 21613 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.729333 21615 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:31.730647 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:31.731264 21336 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 0.
I20251215 08:58:31.736322 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:31.736872 21259 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.742941 21413 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20251215 08:58:31.743325 21413 pending_rounds.cc:85] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Aborting all ops after (but not including) 0
I20251215 08:58:31.743597 21413 pending_rounds.cc:107] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251215 08:58:31.743896 21413 raft_consensus.cc:2889] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:58:31.745467 21885 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:31.746496 21613 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:31.747570 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:31.748128 21336 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.751303 21259 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:31.752364 21381 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:31.754875 21615 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 08:58:31.755208 21615 raft_consensus.cc:697] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:31.755805 21615 consensus_queue.cc:237] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.739046 21164 catalog_manager.cc:5654] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:31.760046 21336 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:31.764302 21457 tablet_bootstrap.cc:492] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:31.764742 21457 ts_tablet_manager.cc:1403] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.142s user 0.031s sys 0.054s
I20251215 08:58:31.767038 21457 raft_consensus.cc:359] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.767736 21457 raft_consensus.cc:385] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:31.767989 21457 raft_consensus.cc:740] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:31.768800 21457 consensus_queue.cc:260] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.771013 21457 ts_tablet_manager.cc:1434] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251215 08:58:31.772023 21457 tablet_bootstrap.cc:492] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Bootstrap starting.
I20251215 08:58:31.777637 21457 tablet_bootstrap.cc:654] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Neither blocks nor log segments found. Creating new log.
I20251215 08:58:31.781276 21162 catalog_manager.cc:5654] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:31.782819 21457 tablet_bootstrap.cc:492] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: No bootstrap required, opened a new log
I20251215 08:58:31.783249 21457 ts_tablet_manager.cc:1403] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20251215 08:58:31.785888 21457 raft_consensus.cc:359] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.786477 21457 raft_consensus.cc:385] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 08:58:31.786743 21457 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Initialized, Role: FOLLOWER
I20251215 08:58:31.790879 21457 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.793108 21457 ts_tablet_manager.cc:1434] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20251215 08:58:31.798944 21163 catalog_manager.cc:5654] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:31.816897 21615 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:31.823736 21164 catalog_manager.cc:5654] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:31.836283 21613 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:31.964020 21858 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:31.965191 21858 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:31.968308 21407 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.970216 21915 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:31.963112 21413 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 08:58:31.992211 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:31.997627 21907 consensus_queue.cc:1048] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:32.005023 21894 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:32.006439 21858 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:32.012588 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:32.015475 21379 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:32.016912 21790 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 08:58:32.034157 21889 consensus_queue.cc:1048] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 08:58:32.177973 21858 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:32.188354 21858 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251215 08:58:33.659700 20109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 08:58:33.694906 21935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 08:58:33.695732 21934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:33.697161 20109 server_base.cc:1047] running on GCE node
W20251215 08:58:33.697552 21937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 08:58:33.699081 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 08:58:33.699301 20109 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 08:58:33.699429 20109 hybrid_clock.cc:648] HybridClock initialized: now 1765789113699418 us; error 0 us; skew 500 ppm
I20251215 08:58:33.699908 20109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251215 08:58:33.702872 20109 webserver.cc:492] Webserver started at http://127.19.163.68:46499/ using document root <none> and password file <none>
I20251215 08:58:33.703414 20109 fs_manager.cc:362] Metadata directory not provided
I20251215 08:58:33.703753 20109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 08:58:33.704069 20109 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 08:58:33.705435 20109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-3-root/instance:
uuid: "f0824fa5450345719cac22fc6ae3ce2e"
format_stamp: "Formatted at 2025-12-15 08:58:33 on dist-test-slave-s29j"
I20251215 08:58:33.711365 20109 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251215 08:58:33.714632 21942 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:33.715524 20109 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251215 08:58:33.715835 20109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-3-root
uuid: "f0824fa5450345719cac22fc6ae3ce2e"
format_stamp: "Formatted at 2025-12-15 08:58:33 on dist-test-slave-s29j"
I20251215 08:58:33.716363 20109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 08:58:33.738925 20109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 08:58:33.740289 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 08:58:33.741917 20109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 08:58:33.745175 20109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 08:58:33.745517 20109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 08:58:33.745820 20109 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 08:58:33.746093 20109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251215 08:58:33.815152 20109 rpc_server.cc:307] RPC server started. Bound to: 127.19.163.68:45831
I20251215 08:58:33.815303 22004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.163.68:45831 every 8 connection(s)
I20251215 08:58:33.819252 20109 auto_leader_rebalancer-test.cc:417] add a tserver: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:33.833771 22005 heartbeater.cc:344] Connected to a master server at 127.19.163.126:33983
I20251215 08:58:33.834184 22005 heartbeater.cc:461] Registering TS with master...
I20251215 08:58:33.834965 22005 heartbeater.cc:507] Master 127.19.163.126:33983 requested a full tablet report, sending...
I20251215 08:58:33.837188 21163 ts_manager.cc:194] Registered new tserver with Master: f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:33.838949 21163 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45538
I20251215 08:58:34.657300 21258 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:34.663805 21412 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.664731 21332 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.666546 21885 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:34.668828 21907 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:34.675839 21885 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.677719 21412 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.679234 21332 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
W20251215 08:58:34.687104 21229 consensus_peers.cc:597] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd5a10a353d64f5c826c27e4a01f6132. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:34.698257 21163 catalog_manager.cc:5654] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:34.708261 21412 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:34.716677 21332 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.719372 22010 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:34.720582 21258 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.723446 22010 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:34.737033 21613 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.742671 21258 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.746057 21332 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.755411 21258 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:34.759409 21164 catalog_manager.cc:5654] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251215 08:58:34.765811 21378 consensus_peers.cc:597] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9582bb1875204089bca2e38a3ff426b3. This is attempt 1: this message will repeat every 5th retry.
W20251215 08:58:34.772676 21229 consensus_peers.cc:597] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 080eec653ea04179a0dceafb6e3ffe47. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:34.775533 21412 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.777513 21332 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.779223 21907 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:34.780479 22009 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:34.795153 21907 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.796676 21412 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.803575 21332 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.815251 21163 catalog_manager.cc:5654] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.831107 21332 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:34.840315 21412 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.841125 22005 heartbeater.cc:499] Master 127.19.163.126:33983 was elected leader, sending a full tablet report...
I20251215 08:58:34.842075 21915 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:34.843140 21258 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:34.845012 21915 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:34.854871 21916 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.862576 21413 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:34.879071 21162 catalog_manager.cc:5654] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
W20251215 08:58:34.883162 21304 consensus_peers.cc:597] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15d0087ada21421eacebc646b271e145. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:34.892911 21258 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:35.080415 22037 ts_tablet_manager.cc:933] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:35.083995 22037 tablet_copy_client.cc:323] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:35.217290 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet dd5a10a353d64f5c826c27e4a01f6132 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:35.217988 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet dd5a10a353d64f5c826c27e4a01f6132 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-dd5a10a353d64f5c826c27e4a01f6132
I20251215 08:58:35.228080 21276 tablet_copy_source_session.cc:215] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:35.233356 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5a10a353d64f5c826c27e4a01f6132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:35.252427 22041 ts_tablet_manager.cc:933] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:35.258059 22037 tablet_copy_client.cc:806] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:35.259037 22037 tablet_copy_client.cc:670] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:35.260715 22043 ts_tablet_manager.cc:933] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:35.265460 22037 tablet_copy_client.cc:538] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:35.277173 22041 tablet_copy_client.cc:323] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:35.281628 22037 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:35.282851 22043 tablet_copy_client.cc:323] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:35.285677 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 080eec653ea04179a0dceafb6e3ffe47 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:35.286227 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 080eec653ea04179a0dceafb6e3ffe47 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-080eec653ea04179a0dceafb6e3ffe47
I20251215 08:58:35.294557 21276 tablet_copy_source_session.cc:215] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:35.299402 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080eec653ea04179a0dceafb6e3ffe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:35.304415 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet 9582bb1875204089bca2e38a3ff426b3 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:35.304936 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet 9582bb1875204089bca2e38a3ff426b3 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-9582bb1875204089bca2e38a3ff426b3
I20251215 08:58:35.314467 22037 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:35.315415 22037 tablet_bootstrap.cc:492] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:35.316080 22037 ts_tablet_manager.cc:1403] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.012s
I20251215 08:58:35.319284 22037 raft_consensus.cc:359] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.320115 22037 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:35.320638 22043 tablet_copy_client.cc:806] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:35.321205 22043 tablet_copy_client.cc:670] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:35.321077 22037 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.322012 21425 tablet_copy_source_session.cc:215] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:35.326195 22043 tablet_copy_client.cc:538] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:35.326994 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9582bb1875204089bca2e38a3ff426b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:35.334074 22043 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:35.344962 22037 ts_tablet_manager.cc:1434] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.029s user 0.013s sys 0.010s
I20251215 08:58:35.348418 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-dd5a10a353d64f5c826c27e4a01f6132 received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:35.349056 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-dd5a10a353d64f5c826c27e4a01f6132 on tablet dd5a10a353d64f5c826c27e4a01f6132 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:35.371600 22041 tablet_copy_client.cc:806] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:35.372309 22041 tablet_copy_client.cc:670] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:35.376235 22041 tablet_copy_client.cc:538] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:35.382320 22043 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:35.383388 22043 tablet_bootstrap.cc:492] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:35.384096 22043 ts_tablet_manager.cc:1403] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.022s
I20251215 08:58:35.384523 22041 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:35.386803 22043 raft_consensus.cc:359] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.387694 22043 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:35.388685 22043 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.391153 22043 ts_tablet_manager.cc:1434] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20251215 08:58:35.394184 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-080eec653ea04179a0dceafb6e3ffe47 received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:35.394556 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-080eec653ea04179a0dceafb6e3ffe47 on tablet 080eec653ea04179a0dceafb6e3ffe47 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:35.406080 22041 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:35.406920 22041 tablet_bootstrap.cc:492] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:35.407469 22041 ts_tablet_manager.cc:1403] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20251215 08:58:35.409802 22041 raft_consensus.cc:359] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.411293 22041 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:35.411926 22041 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.414250 22041 ts_tablet_manager.cc:1434] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251215 08:58:35.415584 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-9582bb1875204089bca2e38a3ff426b3 received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:35.416004 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-9582bb1875204089bca2e38a3ff426b3 on tablet 9582bb1875204089bca2e38a3ff426b3 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:35.457536 22041 ts_tablet_manager.cc:933] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:35.461458 22041 tablet_copy_client.cc:323] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:35.485648 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet 15d0087ada21421eacebc646b271e145 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:35.486179 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet 15d0087ada21421eacebc646b271e145 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-15d0087ada21421eacebc646b271e145
I20251215 08:58:35.492197 21351 tablet_copy_source_session.cc:215] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:35.494700 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d0087ada21421eacebc646b271e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:35.507134 22041 tablet_copy_client.cc:806] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:35.508405 22041 tablet_copy_client.cc:670] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:35.511641 22041 tablet_copy_client.cc:538] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:35.518529 22041 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:35.535411 22041 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:35.536235 22041 tablet_bootstrap.cc:492] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:35.536739 22041 ts_tablet_manager.cc:1403] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.012s
I20251215 08:58:35.539038 22041 raft_consensus.cc:359] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.539788 22041 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:35.540266 22041 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:35.542431 22041 ts_tablet_manager.cc:1434] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251215 08:58:35.543823 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-15d0087ada21421eacebc646b271e145 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:35.544247 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-15d0087ada21421eacebc646b271e145 on tablet 15d0087ada21421eacebc646b271e145 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:35.553309 21980 raft_consensus.cc:1217] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:35.643457 21980 raft_consensus.cc:1217] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:35.809911 21980 raft_consensus.cc:1217] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:35.846989 21980 raft_consensus.cc:1217] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:36.050163 21885 raft_consensus.cc:1064] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:36.052524 21885 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.059157 21412 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.059602 21980 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.060607 21907 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.062007 21907 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.066018 21341 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.067240 21907 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.080806 21907 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.083451 22028 raft_consensus.cc:1064] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:36.082769 21341 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.086647 22028 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.091701 21407 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.103600 21979 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.103600 21980 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.111747 21885 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.112330 21162 catalog_manager.cc:5654] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:36.123515 21407 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.124992 21885 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.130323 21341 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.132280 22059 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.135694 21885 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.137086 21979 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:58:36.142051 21362 debug-util.cc:398] Leaking SignalData structure 0x7b08001969e0 after lost signal to thread 20112
W20251215 08:58:36.143046 21362 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2360 after lost signal to thread 21198
W20251215 08:58:36.147603 21362 debug-util.cc:398] Leaking SignalData structure 0x7b08002d3020 after lost signal to thread 22004
I20251215 08:58:36.154744 21412 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.169695 21341 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.196774 21162 catalog_manager.cc:5654] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.198294 21266 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.207170 21979 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.208045 21407 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.209640 21907 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:36.210608 22058 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:36.219409 22058 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.227523 21409 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.231129 21979 raft_consensus.cc:2955] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.232082 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 080eec653ea04179a0dceafb6e3ffe47 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:36.251083 21266 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.255501 21164 catalog_manager.cc:5654] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:36.264564 21980 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.271936 22058 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:36.275683 21341 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.277280 22028 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:36.277885 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 080eec653ea04179a0dceafb6e3ffe47 with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:36.279202 22028 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.280783 21980 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.312016 21161 catalog_manager.cc:5654] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.321992 22067 tablet_replica.cc:333] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:36.323477 22067 raft_consensus.cc:2243] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:36.325599 22067 raft_consensus.cc:2272] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:36.331007 21400 tablet_service.cc:1552] Processing DeleteTablet for tablet dd5a10a353d64f5c826c27e4a01f6132 with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:36.332093 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet dd5a10a353d64f5c826c27e4a01f6132 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:36.341635 21613 raft_consensus.cc:1064] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:36.344144 21613 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.345587 21336 raft_consensus.cc:2955] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.356972 22070 tablet_replica.cc:333] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:36.364176 22067 ts_tablet_manager.cc:1916] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:36.372330 21266 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.373294 21980 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.373955 22068 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:36.379462 21332 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.382562 22072 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:36.384447 22069 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:36.385797 22070 raft_consensus.cc:2243] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:36.392685 22070 raft_consensus.cc:2272] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:36.396023 22070 ts_tablet_manager.cc:1916] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:36.398725 22067 ts_tablet_manager.cc:1929] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:36.399236 22067 log.cc:1199] T 080eec653ea04179a0dceafb6e3ffe47 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/080eec653ea04179a0dceafb6e3ffe47
I20251215 08:58:36.401536 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 080eec653ea04179a0dceafb6e3ffe47 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:36.426400 22069 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.434111 21980 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.440646 22070 ts_tablet_manager.cc:1929] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:36.441176 22070 log.cc:1199] T dd5a10a353d64f5c826c27e4a01f6132 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/dd5a10a353d64f5c826c27e4a01f6132
I20251215 08:58:36.442353 21266 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.445329 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet dd5a10a353d64f5c826c27e4a01f6132 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:36.455536 21164 catalog_manager.cc:5654] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:36.459472 21336 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.484586 22040 raft_consensus.cc:1064] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:36.487360 22040 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.506606 21409 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.507094 21980 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.509298 21916 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.510851 21915 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:36.512508 21409 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.522246 21336 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.525887 21259 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:36.529058 21916 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:36.534135 22069 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20251215 08:58:36.545601 21979 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.547501 22068 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:36.574841 22068 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.576267 21979 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.579427 22040 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.587656 21412 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.589028 21266 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.589499 21161 catalog_manager.cc:5654] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.592006 21332 raft_consensus.cc:2955] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.602820 21164 catalog_manager.cc:5654] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:36.611058 21979 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.615000 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet 9582bb1875204089bca2e38a3ff426b3 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:36.616346 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 9582bb1875204089bca2e38a3ff426b3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:36.638706 22080 tablet_replica.cc:333] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:36.643340 22080 raft_consensus.cc:2243] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:36.645023 22080 raft_consensus.cc:2272] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:36.653054 21332 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:36.659899 21979 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.661437 21916 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:36.662811 22080 ts_tablet_manager.cc:1916] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:36.670270 21412 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:36.670612 21916 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.672890 21979 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.675546 22040 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:36.679862 21412 raft_consensus.cc:2955] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.694016 21162 catalog_manager.cc:5654] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:36.706178 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet 15d0087ada21421eacebc646b271e145 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:36.712450 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 15d0087ada21421eacebc646b271e145 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:36.714237 22080 ts_tablet_manager.cc:1929] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:36.714735 22080 log.cc:1199] T 9582bb1875204089bca2e38a3ff426b3 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/9582bb1875204089bca2e38a3ff426b3
I20251215 08:58:36.721621 22080 tablet_replica.cc:333] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:36.722327 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet 9582bb1875204089bca2e38a3ff426b3 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:36.723344 22080 raft_consensus.cc:2243] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:36.724510 22080 raft_consensus.cc:2272] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:36.730173 22080 ts_tablet_manager.cc:1916] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:36.754022 22080 ts_tablet_manager.cc:1929] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:36.754531 22080 log.cc:1199] T 15d0087ada21421eacebc646b271e145 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/15d0087ada21421eacebc646b271e145
I20251215 08:58:36.755995 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet 15d0087ada21421eacebc646b271e145 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:37.825451 21259 consensus_queue.cc:237] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
W20251215 08:58:37.833361 21229 consensus_peers.cc:597] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79790fd760ff4ae1b3879326decc531a. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:37.834510 21412 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:37.834760 21341 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:37.835985 21907 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:37.838027 21907 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:37.847424 21885 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:37.851055 21341 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:37.859998 21409 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:37.867548 21332 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:37.870066 21163 catalog_manager.cc:5654] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:37.908046 21412 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:37.910254 21304 consensus_peers.cc:597] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c61929339a494e3fb5f327784d94d817. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:37.910564 22081 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:37.911181 21266 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:37.912739 22081 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:37.922860 22081 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:37.925280 21412 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:37.961244 21259 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:37.985285 21261 consensus_queue.cc:237] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:37.988139 21161 catalog_manager.cc:5654] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251215 08:58:38.010921 21229 consensus_peers.cc:597] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba393161b7454821b7584a44b64bf6b2. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:38.012355 21336 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:38.012418 21409 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:38.013657 21907 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:38.014652 21885 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:38.035209 22082 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:38.038913 21412 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:38.045647 21336 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:38.057252 21161 catalog_manager.cc:5654] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:38.074285 21412 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.086675 21341 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:38.088291 22068 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:38.090257 21261 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:38.091496 22068 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251215 08:58:38.093127 21378 consensus_peers.cc:597] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cdd8fe5ea144c7897b7cf7eaf77c19a. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:38.100430 22084 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:38.102731 21259 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:38.106884 21341 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:38.121124 21161 catalog_manager.cc:5654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:38.286438 22103 ts_tablet_manager.cc:933] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:38.288801 22103 tablet_copy_client.cc:323] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:38.290966 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 79790fd760ff4ae1b3879326decc531a from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:38.291554 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 79790fd760ff4ae1b3879326decc531a from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-79790fd760ff4ae1b3879326decc531a
I20251215 08:58:38.298133 21276 tablet_copy_source_session.cc:215] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:38.300962 22103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79790fd760ff4ae1b3879326decc531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:38.316118 22103 tablet_copy_client.cc:806] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:38.318249 22103 tablet_copy_client.cc:670] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:38.323956 22103 tablet_copy_client.cc:538] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:38.332696 22103 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:38.364624 22103 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:38.365734 22103 tablet_bootstrap.cc:492] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:38.366480 22103 ts_tablet_manager.cc:1403] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.005s
I20251215 08:58:38.369640 22103 raft_consensus.cc:359] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.370524 22103 raft_consensus.cc:740] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:38.371244 22103 consensus_queue.cc:260] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.381335 22103 ts_tablet_manager.cc:1434] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20251215 08:58:38.383996 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-79790fd760ff4ae1b3879326decc531a received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:38.384443 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-79790fd760ff4ae1b3879326decc531a on tablet 79790fd760ff4ae1b3879326decc531a with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:38.491840 22103 ts_tablet_manager.cc:933] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:38.511381 22103 tablet_copy_client.cc:323] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:38.513223 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet ba393161b7454821b7584a44b64bf6b2 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:38.513780 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet ba393161b7454821b7584a44b64bf6b2 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-ba393161b7454821b7584a44b64bf6b2
I20251215 08:58:38.523590 21276 tablet_copy_source_session.cc:215] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:38.539000 22103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba393161b7454821b7584a44b64bf6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:38.540462 22108 ts_tablet_manager.cc:933] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:38.568614 22108 tablet_copy_client.cc:323] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:38.573627 22103 tablet_copy_client.cc:806] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:38.576592 22103 tablet_copy_client.cc:670] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:38.574729 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet c61929339a494e3fb5f327784d94d817 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:38.578213 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet c61929339a494e3fb5f327784d94d817 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-c61929339a494e3fb5f327784d94d817
I20251215 08:58:38.589879 22110 ts_tablet_manager.cc:933] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:38.590375 22103 tablet_copy_client.cc:538] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:38.602084 22103 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:38.625429 22110 tablet_copy_client.cc:323] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:38.627065 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:38.627620 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-0cdd8fe5ea144c7897b7cf7eaf77c19a
I20251215 08:58:38.636097 21351 tablet_copy_source_session.cc:215] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:38.637149 21425 tablet_copy_source_session.cc:215] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:38.643078 22110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:38.650945 22108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61929339a494e3fb5f327784d94d817. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:38.661651 22110 tablet_copy_client.cc:806] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:38.662369 22110 tablet_copy_client.cc:670] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:38.666275 22110 tablet_copy_client.cc:538] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:38.668437 22103 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:38.669442 22103 tablet_bootstrap.cc:492] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:38.670498 22103 ts_tablet_manager.cc:1403] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.069s user 0.024s sys 0.007s
I20251215 08:58:38.675094 22103 raft_consensus.cc:359] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.676482 22103 raft_consensus.cc:740] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:38.677938 22103 consensus_queue.cc:260] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.682739 22103 ts_tablet_manager.cc:1434] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20251215 08:58:38.685493 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-ba393161b7454821b7584a44b64bf6b2 received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:38.685894 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-ba393161b7454821b7584a44b64bf6b2 on tablet ba393161b7454821b7584a44b64bf6b2 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:38.696552 22110 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:38.704121 22108 tablet_copy_client.cc:806] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:38.708045 22108 tablet_copy_client.cc:670] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:38.711956 22108 tablet_copy_client.cc:538] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:38.728293 22108 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:38.741159 22110 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:38.742348 22110 tablet_bootstrap.cc:492] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:38.743182 22110 ts_tablet_manager.cc:1403] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.062s user 0.028s sys 0.004s
I20251215 08:58:38.746399 22110 raft_consensus.cc:359] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.747545 22110 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:38.748320 22110 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.757373 22108 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:38.760411 22108 tablet_bootstrap.cc:492] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:38.761294 22108 ts_tablet_manager.cc:1403] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.004s
I20251215 08:58:38.761884 22110 ts_tablet_manager.cc:1434] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20251215 08:58:38.764163 22108 raft_consensus.cc:359] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.765146 22108 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:38.765859 22108 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:38.768824 22108 ts_tablet_manager.cc:1434] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20251215 08:58:38.771903 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-c61929339a494e3fb5f327784d94d817 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:38.772349 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-c61929339a494e3fb5f327784d94d817 on tablet c61929339a494e3fb5f327784d94d817 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:38.774173 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-0cdd8fe5ea144c7897b7cf7eaf77c19a received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:38.774529 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-0cdd8fe5ea144c7897b7cf7eaf77c19a on tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:38.889276 21979 raft_consensus.cc:1217] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:39.076865 21979 raft_consensus.cc:1217] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:39.142004 21979 raft_consensus.cc:1217] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:39.201618 21980 raft_consensus.cc:1217] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:39.344937 22083 raft_consensus.cc:1064] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:39.347385 22083 consensus_queue.cc:237] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:39.364276 21980 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.365391 21336 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.367693 21885 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.370299 22115 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.381157 21415 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.382898 21885 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.413595 21885 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.415158 21336 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.429718 21980 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.436683 21162 catalog_manager.cc:5654] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.439363 21409 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.492472 21261 consensus_queue.cc:237] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:39.504516 21980 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:39.506698 22115 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20251215 08:58:39.524394 22083 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.531217 21980 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.534770 21412 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:39.537269 22115 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:39.550331 21412 raft_consensus.cc:2955] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.554915 21164 catalog_manager.cc:5654] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.557996 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 79790fd760ff4ae1b3879326decc531a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:39.573185 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 79790fd760ff4ae1b3879326decc531a with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:39.591216 22126 tablet_replica.cc:333] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:39.593773 22126 raft_consensus.cc:2243] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:39.594750 22126 raft_consensus.cc:2272] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:39.599934 22126 ts_tablet_manager.cc:1916] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:39.607939 22040 raft_consensus.cc:1064] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:39.610443 22040 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:39.620227 21885 raft_consensus.cc:1064] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:39.642153 22126 ts_tablet_manager.cc:1929] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:39.623345 21885 consensus_queue.cc:237] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:39.650020 22126 log.cc:1199] T 79790fd760ff4ae1b3879326decc531a P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/79790fd760ff4ae1b3879326decc531a
I20251215 08:58:39.651772 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 79790fd760ff4ae1b3879326decc531a (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:39.653352 22117 raft_consensus.cc:1064] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:39.655128 22117 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:39.647301 21409 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.648075 21980 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.649055 21266 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.659296 22040 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:39.661212 22081 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.662577 22040 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.681031 21336 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.683126 22068 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.689914 21266 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.690176 21979 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.689699 22081 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.691128 22068 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.694437 22117 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.702240 21412 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.730829 21164 catalog_manager.cc:5654] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.739506 21258 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.743526 22116 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.745601 21980 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.747328 21409 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.751856 22115 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.753618 22115 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.753820 21341 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:39.748853 21978 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.763325 21261 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.783367 21885 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:39.806094 21336 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:39.809487 21980 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.824241 21258 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:39.828979 21164 catalog_manager.cc:5654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:39.838212 22081 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:39.837347 21333 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.867609 21977 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:39.871083 22136 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:39.876580 22136 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.886687 21258 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.886430 22133 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.892436 21409 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:39.906113 21335 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:39.908360 21979 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:39.917568 22140 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:39.924120 21979 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.926334 22140 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:39.902022 21977 raft_consensus.cc:2955] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.930135 22138 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.935911 21336 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.938809 21332 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.977511 21161 catalog_manager.cc:5654] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:39.984597 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet c61929339a494e3fb5f327784d94d817 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:39.990638 21415 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:39.993695 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:40.002622 21162 catalog_manager.cc:5654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:39.980338 21979 raft_consensus.cc:2955] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:40.019253 21258 consensus_queue.cc:237] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:40.026588 21979 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:40.027792 21332 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:40.031608 22141 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:58:40.032585 21885 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:40.036289 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:40.049460 21336 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:40.038401 21161 catalog_manager.cc:5654] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:40.043943 21885 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:40.082351 22146 tablet_replica.cc:333] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:40.084261 22146 raft_consensus.cc:2243] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:40.085155 22146 raft_consensus.cc:2272] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:40.086232 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet ba393161b7454821b7584a44b64bf6b2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:40.087759 21400 tablet_service.cc:1552] Processing DeleteTablet for tablet c61929339a494e3fb5f327784d94d817 with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:40.091784 22146 ts_tablet_manager.cc:1916] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:40.083870 21979 raft_consensus.cc:2955] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:40.100672 21164 catalog_manager.cc:5654] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:40.110898 22146 ts_tablet_manager.cc:1929] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:40.111457 22146 log.cc:1199] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/0cdd8fe5ea144c7897b7cf7eaf77c19a
I20251215 08:58:40.113350 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet 0cdd8fe5ea144c7897b7cf7eaf77c19a (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:40.131850 21402 tablet_service.cc:1552] Processing DeleteTablet for tablet ba393161b7454821b7584a44b64bf6b2 with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:40.133227 22147 tablet_replica.cc:333] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:40.134210 22147 raft_consensus.cc:2243] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:40.135002 22147 raft_consensus.cc:2272] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:40.138152 22147 ts_tablet_manager.cc:1916] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:40.156297 22147 ts_tablet_manager.cc:1929] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:40.156829 22147 log.cc:1199] T c61929339a494e3fb5f327784d94d817 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/c61929339a494e3fb5f327784d94d817
I20251215 08:58:40.158532 22147 tablet_replica.cc:333] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:40.158599 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet c61929339a494e3fb5f327784d94d817 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:40.163655 22147 raft_consensus.cc:2243] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:40.164530 22147 raft_consensus.cc:2272] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:40.168236 22147 ts_tablet_manager.cc:1916] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:40.185854 22147 ts_tablet_manager.cc:1929] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:40.186277 22147 log.cc:1199] T ba393161b7454821b7584a44b64bf6b2 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/ba393161b7454821b7584a44b64bf6b2
I20251215 08:58:40.187776 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet ba393161b7454821b7584a44b64bf6b2 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:41.155661 21409 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.162041 21258 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:41.162335 21378 consensus_peers.cc:597] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c3a05444c334cad8e7562a15e4d68ba. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:41.162922 21333 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:41.163287 22150 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:41.164073 22138 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:41.170872 22117 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.172132 21336 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.176030 21258 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.186962 21164 catalog_manager.cc:5654] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.194146 21266 consensus_queue.cc:237] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
W20251215 08:58:41.204138 21229 consensus_peers.cc:597] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d471c9f4eb74ea3a20cec86631e4ab0. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:41.205319 21409 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:41.207715 22141 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:41.215260 21336 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:41.217298 22153 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:41.234964 22141 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.239625 21336 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.241791 21409 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.256949 21336 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
W20251215 08:58:41.266517 21304 consensus_peers.cc:597] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d1d9a9be078441587cf5da752d59c33. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:41.268075 21409 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:41.271858 21163 catalog_manager.cc:5654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.273622 22145 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:41.275632 21258 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:41.281289 22145 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:41.287714 22159 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.294785 21409 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.300918 21415 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.303763 21266 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.307585 21163 catalog_manager.cc:5654] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:41.326747 21333 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:41.326849 21378 consensus_peers.cc:597] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c50acc4d0854bd08c6afc283d478ca7. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:41.327811 21258 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:41.328269 22138 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:41.329151 22117 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:41.342919 22117 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.347412 21336 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.354421 21258 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.366019 21164 catalog_manager.cc:5654] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:41.631867 22170 ts_tablet_manager.cc:933] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:41.633626 22170 tablet_copy_client.cc:323] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:41.634887 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet 0c3a05444c334cad8e7562a15e4d68ba from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:41.635366 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet 0c3a05444c334cad8e7562a15e4d68ba from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-0c3a05444c334cad8e7562a15e4d68ba
I20251215 08:58:41.642321 21425 tablet_copy_source_session.cc:215] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:41.644940 22170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c3a05444c334cad8e7562a15e4d68ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:41.657588 22170 tablet_copy_client.cc:806] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:41.658118 22170 tablet_copy_client.cc:670] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:41.661423 22170 tablet_copy_client.cc:538] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:41.669529 22170 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:41.687686 22170 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:41.688341 22170 tablet_bootstrap.cc:492] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:41.688969 22170 ts_tablet_manager.cc:1403] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20251215 08:58:41.691393 22170 raft_consensus.cc:359] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.692170 22170 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:41.692662 22170 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.699034 22170 ts_tablet_manager.cc:1434] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251215 08:58:41.700593 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-0c3a05444c334cad8e7562a15e4d68ba received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:41.700999 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-0c3a05444c334cad8e7562a15e4d68ba on tablet 0c3a05444c334cad8e7562a15e4d68ba with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:41.772930 22170 ts_tablet_manager.cc:933] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:41.775162 22170 tablet_copy_client.cc:323] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:41.776686 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet 4c50acc4d0854bd08c6afc283d478ca7 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:41.777284 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet 4c50acc4d0854bd08c6afc283d478ca7 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-4c50acc4d0854bd08c6afc283d478ca7
I20251215 08:58:41.784032 21425 tablet_copy_source_session.cc:215] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:41.786826 22170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c50acc4d0854bd08c6afc283d478ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:41.800894 22170 tablet_copy_client.cc:806] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:41.801647 22170 tablet_copy_client.cc:670] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:41.804984 22170 tablet_copy_client.cc:538] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:41.814604 22170 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:41.828382 22176 ts_tablet_manager.cc:933] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:41.829704 22175 ts_tablet_manager.cc:933] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:41.837147 22176 tablet_copy_client.cc:323] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:41.838979 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 3d471c9f4eb74ea3a20cec86631e4ab0 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:41.839497 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 3d471c9f4eb74ea3a20cec86631e4ab0 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-3d471c9f4eb74ea3a20cec86631e4ab0
I20251215 08:58:41.841548 22175 tablet_copy_client.cc:323] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:41.848325 21276 tablet_copy_source_session.cc:215] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:41.851835 22170 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:41.851301 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet 4d1d9a9be078441587cf5da752d59c33 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:41.852419 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet 4d1d9a9be078441587cf5da752d59c33 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-4d1d9a9be078441587cf5da752d59c33
I20251215 08:58:41.854084 22176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d471c9f4eb74ea3a20cec86631e4ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:41.864614 21351 tablet_copy_source_session.cc:215] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:41.866776 22170 tablet_bootstrap.cc:492] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:41.867563 22170 ts_tablet_manager.cc:1403] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.053s user 0.022s sys 0.008s
I20251215 08:58:41.870361 22170 raft_consensus.cc:359] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.871268 22170 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:41.871956 22170 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.874416 22170 ts_tablet_manager.cc:1434] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251215 08:58:41.876869 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-4c50acc4d0854bd08c6afc283d478ca7 received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:41.877359 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-4c50acc4d0854bd08c6afc283d478ca7 on tablet 4c50acc4d0854bd08c6afc283d478ca7 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:41.877619 22175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d1d9a9be078441587cf5da752d59c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:41.881045 22176 tablet_copy_client.cc:806] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:41.882046 22176 tablet_copy_client.cc:670] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:41.889478 22176 tablet_copy_client.cc:538] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:41.898555 22176 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:41.898509 22175 tablet_copy_client.cc:806] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:41.899396 22175 tablet_copy_client.cc:670] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:41.904631 22175 tablet_copy_client.cc:538] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:41.918371 22175 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:41.927825 22176 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:41.928982 22176 tablet_bootstrap.cc:492] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:41.929756 22176 ts_tablet_manager.cc:1403] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.008s
I20251215 08:58:41.932675 22176 raft_consensus.cc:359] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.933688 22176 raft_consensus.cc:740] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:41.934406 22176 consensus_queue.cc:260] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.939831 22176 ts_tablet_manager.cc:1434] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251215 08:58:41.942776 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-3d471c9f4eb74ea3a20cec86631e4ab0 received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:41.943318 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-3d471c9f4eb74ea3a20cec86631e4ab0 on tablet 3d471c9f4eb74ea3a20cec86631e4ab0 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:41.961331 22175 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:41.962268 22175 tablet_bootstrap.cc:492] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:41.962914 22175 ts_tablet_manager.cc:1403] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.008s
I20251215 08:58:41.965467 22175 raft_consensus.cc:359] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.966362 22175 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:41.967111 22175 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:41.969440 22175 ts_tablet_manager.cc:1434] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251215 08:58:41.971107 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-4d1d9a9be078441587cf5da752d59c33 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:41.971500 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-4d1d9a9be078441587cf5da752d59c33 on tablet 4d1d9a9be078441587cf5da752d59c33 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:42.248253 21979 raft_consensus.cc:1217] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:42.256562 21979 raft_consensus.cc:1217] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:42.329896 21979 raft_consensus.cc:1217] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:42.434088 21977 raft_consensus.cc:1217] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:42.734943 22138 raft_consensus.cc:1064] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:42.737126 22138 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:42.744335 21258 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.744587 21977 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.745757 22138 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:42.746001 21333 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.746790 22169 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:42.748342 22183 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:42.766202 22138 raft_consensus.cc:1064] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:42.769937 22184 raft_consensus.cc:1064] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:42.770437 22138 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:42.772732 22184 consensus_queue.cc:237] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:42.779870 22169 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.781117 21979 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.783361 22186 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:42.782629 21979 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.786062 21333 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.788269 21415 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.788080 22184 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:42.800221 21164 catalog_manager.cc:5654] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.801640 22186 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:42.824433 21266 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.838119 21258 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.838899 21336 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.840909 22196 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:42.851830 22183 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:42.852674 21979 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:42.857329 22183 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251215 08:58:42.859922 22193 log.cc:927] Time spent T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.051s user 0.000s sys 0.002s
W20251215 08:58:42.866492 22191 log.cc:927] Time spent T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.065s user 0.002s sys 0.000s
I20251215 08:58:42.886406 21332 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.889878 22195 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.893201 21335 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.903040 22196 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.908514 21977 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.915205 21258 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.931277 21979 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.934137 21335 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:42.944181 21412 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.008698 21412 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:43.012082 21161 catalog_manager.cc:5654] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.014178 21161 catalog_manager.cc:5654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.014932 21258 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.015933 21977 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.016243 22169 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:43.018014 22138 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:43.028928 22138 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.031077 21258 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.034497 21977 raft_consensus.cc:2955] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.041589 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 0c3a05444c334cad8e7562a15e4d68ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:43.050880 21258 consensus_queue.cc:237] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:43.058549 22188 raft_consensus.cc:1064] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:43.058650 21979 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.058101 21163 catalog_manager.cc:5654] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.060083 22186 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251215 08:58:43.063311 21151 catalog_manager.cc:5177] ChangeConfig:REMOVE_PEER RPC for tablet 0c3a05444c334cad8e7562a15e4d68ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20251215 08:58:43.063174 22188 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:43.069285 22185 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.072871 21413 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:43.075654 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c3a05444c334cad8e7562a15e4d68ba with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:43.090302 21258 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.089426 21979 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.101214 21332 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.099005 22196 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20251215 08:58:43.112303 21977 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.116281 22184 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:58:43.122601 22211 tablet_replica.cc:333] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:43.124117 22196 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:58:43.124665 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 3d471c9f4eb74ea3a20cec86631e4ab0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:43.121999 21333 raft_consensus.cc:2955] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.126912 22206 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.131973 21162 catalog_manager.cc:5654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.140833 22211 raft_consensus.cc:2243] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:43.151096 22211 raft_consensus.cc:2272] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:43.146572 21261 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.147576 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4c50acc4d0854bd08c6afc283d478ca7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:43.147792 21400 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d471c9f4eb74ea3a20cec86631e4ab0 with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:43.158635 21161 catalog_manager.cc:5654] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:43.163331 21413 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:43.166357 22187 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:43.167061 21979 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:43.172520 22159 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:43.144801 21977 raft_consensus.cc:2955] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.188563 21258 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:43.192410 22215 tablet_replica.cc:333] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:43.193082 22188 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20251215 08:58:43.193614 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c50acc4d0854bd08c6afc283d478ca7 with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:43.209425 22211 ts_tablet_manager.cc:1916] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:43.210402 22215 raft_consensus.cc:2243] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:43.211315 22215 raft_consensus.cc:2272] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:43.215006 22215 ts_tablet_manager.cc:1916] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:43.227968 22211 ts_tablet_manager.cc:1929] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:43.228564 22211 log.cc:1199] T 0c3a05444c334cad8e7562a15e4d68ba P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/0c3a05444c334cad8e7562a15e4d68ba
I20251215 08:58:43.230453 22211 tablet_replica.cc:333] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:43.231598 22211 raft_consensus.cc:2243] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:43.231825 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 0c3a05444c334cad8e7562a15e4d68ba (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:43.232414 22211 raft_consensus.cc:2272] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:43.236819 22211 ts_tablet_manager.cc:1916] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:43.236079 22159 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.246245 21977 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.246526 21412 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.253064 22211 ts_tablet_manager.cc:1929] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:43.253547 22211 log.cc:1199] T 4c50acc4d0854bd08c6afc283d478ca7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/4c50acc4d0854bd08c6afc283d478ca7
I20251215 08:58:43.256016 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 4c50acc4d0854bd08c6afc283d478ca7 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:43.268008 21161 catalog_manager.cc:5654] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.259613 22215 ts_tablet_manager.cc:1929] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:43.271260 22215 log.cc:1199] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/3d471c9f4eb74ea3a20cec86631e4ab0
I20251215 08:58:43.275869 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet 3d471c9f4eb74ea3a20cec86631e4ab0 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:43.281873 21261 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.292958 21332 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:43.301141 21977 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.302703 22188 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:43.306354 21412 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:43.314781 22188 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:58:43.318632 22145 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.323590 21977 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.327629 21412 raft_consensus.cc:2955] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.328506 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4d1d9a9be078441587cf5da752d59c33 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:43.344102 21162 catalog_manager.cc:5654] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:43.353173 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d1d9a9be078441587cf5da752d59c33 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:43.357731 22219 tablet_replica.cc:333] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:43.358682 22219 raft_consensus.cc:2243] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:43.359198 22219 raft_consensus.cc:2272] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:43.362170 22219 ts_tablet_manager.cc:1916] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:43.372961 22219 ts_tablet_manager.cc:1929] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:43.373329 22219 log.cc:1199] T 4d1d9a9be078441587cf5da752d59c33 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/4d1d9a9be078441587cf5da752d59c33
I20251215 08:58:43.374724 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet 4d1d9a9be078441587cf5da752d59c33 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:44.410997 21412 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.417029 21335 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:44.418223 21378 consensus_peers.cc:597] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f800ccb361734ac18102a3502ee639d8. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:44.418220 22138 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:44.419049 21261 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:44.420492 22169 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:44.437853 21335 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.437279 22138 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.442008 21266 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.460173 21161 catalog_manager.cc:5654] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.477226 21412 consensus_queue.cc:237] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.486583 21335 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:44.487635 21378 consensus_peers.cc:597] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d7af00ee6834714a059e045b3c13322. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:44.487845 22228 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:44.488373 21266 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:44.489600 22169 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:44.512106 22138 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.513563 21261 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.523343 21335 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.529752 21333 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.541956 21164 catalog_manager.cc:5654] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:44.546450 21412 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:44.548182 21304 consensus_peers.cc:597] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14cfff8c5a44ad78299ada0ca53baa9. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:44.549202 22188 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:44.550872 21261 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:44.558117 22145 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:44.565867 22238 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.569563 21412 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.578557 21261 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.584210 21335 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.587395 21161 catalog_manager.cc:5654] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:44.602870 21412 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:44.603998 21304 consensus_peers.cc:597] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6e3fc5ec2cc402a96b9fcf2fa3cc5c3. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:44.606139 22238 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:44.610641 21261 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:44.612102 22238 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:44.620222 22188 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.621610 21261 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.621701 21412 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:44.638391 21161 catalog_manager.cc:5654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:44.803969 22243 ts_tablet_manager.cc:933] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:44.805934 22243 tablet_copy_client.cc:323] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:44.808807 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet f800ccb361734ac18102a3502ee639d8 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:44.809401 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet f800ccb361734ac18102a3502ee639d8 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-f800ccb361734ac18102a3502ee639d8
I20251215 08:58:44.818926 21425 tablet_copy_source_session.cc:215] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:44.823280 22243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f800ccb361734ac18102a3502ee639d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:44.839118 22243 tablet_copy_client.cc:806] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:44.839881 22243 tablet_copy_client.cc:670] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:44.844008 22243 tablet_copy_client.cc:538] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:44.851227 22243 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:44.872057 22243 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:44.872853 22243 tablet_bootstrap.cc:492] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:44.873565 22243 ts_tablet_manager.cc:1403] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20251215 08:58:44.875988 22243 raft_consensus.cc:359] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.876699 22243 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:44.877337 22243 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.880810 22243 ts_tablet_manager.cc:1434] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251215 08:58:44.883483 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-f800ccb361734ac18102a3502ee639d8 received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:44.883952 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-f800ccb361734ac18102a3502ee639d8 on tablet f800ccb361734ac18102a3502ee639d8 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:44.891449 22243 ts_tablet_manager.cc:933] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:44.893112 22243 tablet_copy_client.cc:323] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:44.894483 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet 8d7af00ee6834714a059e045b3c13322 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:44.894908 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet 8d7af00ee6834714a059e045b3c13322 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-8d7af00ee6834714a059e045b3c13322
I20251215 08:58:44.901319 21425 tablet_copy_source_session.cc:215] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:44.903837 22243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d7af00ee6834714a059e045b3c13322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:44.917490 22243 tablet_copy_client.cc:806] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:44.918037 22243 tablet_copy_client.cc:670] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:44.921420 22243 tablet_copy_client.cc:538] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:44.928700 22243 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:44.947851 22243 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:44.948798 22243 tablet_bootstrap.cc:492] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:44.950300 22243 ts_tablet_manager.cc:1403] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.004s
I20251215 08:58:44.952770 22243 raft_consensus.cc:359] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.953433 22243 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:44.953984 22243 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:44.958468 22243 ts_tablet_manager.cc:1434] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251215 08:58:44.961419 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-8d7af00ee6834714a059e045b3c13322 received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:44.961768 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-8d7af00ee6834714a059e045b3c13322 on tablet 8d7af00ee6834714a059e045b3c13322 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:45.153685 22243 ts_tablet_manager.cc:933] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:45.157428 22243 tablet_copy_client.cc:323] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:45.160115 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:45.160697 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-a6e3fc5ec2cc402a96b9fcf2fa3cc5c3
I20251215 08:58:45.167217 21351 tablet_copy_source_session.cc:215] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:45.169253 22250 ts_tablet_manager.cc:933] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:45.171662 22243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:45.176841 22250 tablet_copy_client.cc:323] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:45.180701 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet f14cfff8c5a44ad78299ada0ca53baa9 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:45.181545 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet f14cfff8c5a44ad78299ada0ca53baa9 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-f14cfff8c5a44ad78299ada0ca53baa9
I20251215 08:58:45.189149 21351 tablet_copy_source_session.cc:215] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:45.192324 22243 tablet_copy_client.cc:806] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:45.192973 22243 tablet_copy_client.cc:670] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:45.194553 22250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14cfff8c5a44ad78299ada0ca53baa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:45.196642 22243 tablet_copy_client.cc:538] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:45.204962 22243 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:45.224484 22250 tablet_copy_client.cc:806] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:45.225237 22250 tablet_copy_client.cc:670] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:45.236707 22250 tablet_copy_client.cc:538] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:45.244570 22243 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:45.244793 22250 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:45.245615 22243 tablet_bootstrap.cc:492] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:45.248570 22243 ts_tablet_manager.cc:1403] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.044s user 0.025s sys 0.003s
I20251215 08:58:45.251093 22243 raft_consensus.cc:359] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:45.251926 22243 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:45.253374 22243 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:45.257725 22243 ts_tablet_manager.cc:1434] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251215 08:58:45.259392 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:45.259820 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 on tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:45.275676 22250 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:45.276736 22250 tablet_bootstrap.cc:492] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:45.277537 22250 ts_tablet_manager.cc:1403] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.008s
I20251215 08:58:45.280210 22250 raft_consensus.cc:359] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:45.280990 22250 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:45.281602 22250 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:45.283555 22250 ts_tablet_manager.cc:1434] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251215 08:58:45.285511 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-f14cfff8c5a44ad78299ada0ca53baa9 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:45.286068 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-f14cfff8c5a44ad78299ada0ca53baa9 on tablet f14cfff8c5a44ad78299ada0ca53baa9 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:45.394984 21979 raft_consensus.cc:1217] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:45.513162 21979 raft_consensus.cc:1217] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:45.665778 21979 raft_consensus.cc:1217] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:45.774830 21979 raft_consensus.cc:1217] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:46.000057 22138 raft_consensus.cc:1064] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:46.001821 22138 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.007462 21332 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.008245 21266 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.008446 21979 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.009354 22138 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.010473 22228 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.011202 22169 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.028038 22228 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.030751 21979 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.029711 21332 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.044245 21162 catalog_manager.cc:5654] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.045255 21266 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.087198 21412 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.092278 22228 raft_consensus.cc:1064] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:46.095805 22228 consensus_queue.cc:237] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.106938 21335 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.107321 21977 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.108052 21979 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.108085 21333 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.110114 22138 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.111666 22138 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.110072 22169 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.113495 22261 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.118440 21266 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.119933 22228 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.122066 22169 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.124063 21332 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.137221 22238 raft_consensus.cc:1064] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:46.142794 21979 raft_consensus.cc:2955] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:58:46.143484 22255 consensus_queue.cc:1175] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "beefe15185ac42ef9a20251007f39a56" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 2 } }
I20251215 08:58:46.145089 22238 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.147552 21164 catalog_manager.cc:5654] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.151605 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet f800ccb361734ac18102a3502ee639d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:46.165572 22138 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.170717 21978 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.178817 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet f800ccb361734ac18102a3502ee639d8 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:46.198330 21412 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.199796 21266 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.201941 22238 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.202605 21261 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.211371 21161 catalog_manager.cc:5654] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.218606 22256 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:46.222978 22273 tablet_replica.cc:333] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:46.224933 21978 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.226681 21335 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.237555 22238 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.257841 22255 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.261911 22273 raft_consensus.cc:2243] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:46.262823 22273 raft_consensus.cc:2272] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:46.264837 21412 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.265738 21413 consensus_queue.cc:237] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.266295 22273 ts_tablet_manager.cc:1916] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:46.275174 21977 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.280409 21261 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.280565 22228 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.283694 22261 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.287473 21978 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.287909 22169 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.289144 21977 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.298465 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 8d7af00ee6834714a059e045b3c13322 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:46.309053 21266 raft_consensus.cc:2955] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.311079 21161 catalog_manager.cc:5654] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:46.327472 22273 ts_tablet_manager.cc:1929] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:46.328047 22273 log.cc:1199] T f800ccb361734ac18102a3502ee639d8 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/f800ccb361734ac18102a3502ee639d8
I20251215 08:58:46.330139 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet f800ccb361734ac18102a3502ee639d8 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:46.330991 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d7af00ee6834714a059e045b3c13322 with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:46.344872 21164 catalog_manager.cc:5654] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.347657 21259 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.348794 22279 tablet_replica.cc:333] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:46.362285 22238 raft_consensus.cc:1064] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:46.364843 22238 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.377573 22279 raft_consensus.cc:2243] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:46.378438 22279 raft_consensus.cc:2272] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:46.383117 21412 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.383383 21978 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.384399 21259 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:46.384472 22278 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.385372 22255 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.387091 22255 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:46.389321 22279 ts_tablet_manager.cc:1916] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:46.390664 21335 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.396135 21977 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.397485 22255 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.398729 21266 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.400384 22278 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.416558 22279 ts_tablet_manager.cc:1929] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:46.417167 22279 log.cc:1199] T 8d7af00ee6834714a059e045b3c13322 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/8d7af00ee6834714a059e045b3c13322
I20251215 08:58:46.416702 22277 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.420894 22256 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.423168 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 8d7af00ee6834714a059e045b3c13322 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:46.425241 21261 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.426503 21977 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.429674 21979 raft_consensus.cc:2955] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.432484 21266 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.446388 21164 catalog_manager.cc:5654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:46.448123 21413 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.449482 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet f14cfff8c5a44ad78299ada0ca53baa9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:46.473521 21335 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:46.480780 21161 catalog_manager.cc:5654] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.490135 21979 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.490260 21400 tablet_service.cc:1552] Processing DeleteTablet for tablet f14cfff8c5a44ad78299ada0ca53baa9 with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:46.492269 22238 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.493605 21413 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:46.501518 22256 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:46.510986 22255 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.512060 22285 tablet_replica.cc:333] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:46.511629 21977 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.513326 22285 raft_consensus.cc:2243] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:46.514071 22285 raft_consensus.cc:2272] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:46.520268 22285 ts_tablet_manager.cc:1916] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:46.519847 21413 raft_consensus.cc:2955] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:46.526379 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:46.535318 21161 catalog_manager.cc:5654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:46.543066 22285 ts_tablet_manager.cc:1929] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:46.543519 22285 log.cc:1199] T f14cfff8c5a44ad78299ada0ca53baa9 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/f14cfff8c5a44ad78299ada0ca53baa9
I20251215 08:58:46.545256 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet f14cfff8c5a44ad78299ada0ca53baa9 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:46.553160 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:46.556890 22273 tablet_replica.cc:333] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:46.557896 22273 raft_consensus.cc:2243] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:46.558650 22273 raft_consensus.cc:2272] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:46.562314 22273 ts_tablet_manager.cc:1916] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:46.574255 22273 ts_tablet_manager.cc:1929] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:46.574658 22273 log.cc:1199] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/a6e3fc5ec2cc402a96b9fcf2fa3cc5c3
I20251215 08:58:46.575870 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:47.612123 21261 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:47.618067 21412 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:47.618254 21335 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251215 08:58:47.618527 21229 consensus_peers.cc:597] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cf7b0165c00477b9c1b94c6a89052a6. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:47.619334 22287 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:47.620025 22280 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:47.630010 21335 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.630061 22286 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.633229 21412 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.646744 21164 catalog_manager.cc:5654] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 2 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.655980 21335 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:47.664716 21409 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:47.666462 21261 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:47.666848 21304 consensus_peers.cc:597] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 124c9efc80174ca585aaee7e0f8a12f0. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:47.667762 22255 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:47.669111 22256 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:47.690092 22294 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.697167 21261 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.706244 21409 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.723415 21261 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:47.726570 21161 catalog_manager.cc:5654] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251215 08:58:47.730185 21229 consensus_peers.cc:597] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d1675f3765f488e9d58a2ba2993457e. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:47.730434 21335 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:47.730396 21409 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:47.731912 22280 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:47.733985 22280 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:47.750793 22286 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.755388 21409 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.757288 21335 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.764533 21266 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:47.769244 21163 catalog_manager.cc:5654] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251215 08:58:47.783177 21229 consensus_peers.cc:597] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1db12563817b44429a055b7dcdd066c3. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:47.783932 21335 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:47.784373 21412 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:47.785359 22280 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251215 08:58:47.787487 22291 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:47.803300 22280 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.812775 21335 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.826140 21412 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:47.822086 21161 catalog_manager.cc:5654] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:48.091430 22309 ts_tablet_manager.cc:933] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:48.093261 22309 tablet_copy_client.cc:323] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:48.094764 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet 124c9efc80174ca585aaee7e0f8a12f0 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:48.095312 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet 124c9efc80174ca585aaee7e0f8a12f0 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-124c9efc80174ca585aaee7e0f8a12f0
I20251215 08:58:48.102653 21351 tablet_copy_source_session.cc:215] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:48.106228 22309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 124c9efc80174ca585aaee7e0f8a12f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:48.120421 22309 tablet_copy_client.cc:806] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:48.121023 22309 tablet_copy_client.cc:670] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:48.127415 22309 tablet_copy_client.cc:538] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:48.136324 22309 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:48.161341 22309 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:48.162367 22309 tablet_bootstrap.cc:492] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:48.163015 22309 ts_tablet_manager.cc:1403] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20251215 08:58:48.165382 22309 raft_consensus.cc:359] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.166102 22309 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:48.166887 22309 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.169950 22309 ts_tablet_manager.cc:1434] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20251215 08:58:48.172134 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-124c9efc80174ca585aaee7e0f8a12f0 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:48.172492 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-124c9efc80174ca585aaee7e0f8a12f0 on tablet 124c9efc80174ca585aaee7e0f8a12f0 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:48.200330 22309 ts_tablet_manager.cc:933] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:48.201879 22309 tablet_copy_client.cc:323] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:48.203361 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 9d1675f3765f488e9d58a2ba2993457e from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:48.203867 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 9d1675f3765f488e9d58a2ba2993457e from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-9d1675f3765f488e9d58a2ba2993457e
I20251215 08:58:48.211387 21276 tablet_copy_source_session.cc:215] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:48.214547 22309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1675f3765f488e9d58a2ba2993457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:48.229770 22309 tablet_copy_client.cc:806] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:48.230304 22309 tablet_copy_client.cc:670] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:48.237824 22309 tablet_copy_client.cc:538] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:48.240813 22314 ts_tablet_manager.cc:933] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:48.242503 22314 tablet_copy_client.cc:323] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:48.243780 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 3cf7b0165c00477b9c1b94c6a89052a6 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:48.244235 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 3cf7b0165c00477b9c1b94c6a89052a6 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-3cf7b0165c00477b9c1b94c6a89052a6
I20251215 08:58:48.246309 22309 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:48.250749 21276 tablet_copy_source_session.cc:215] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:48.254097 22314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf7b0165c00477b9c1b94c6a89052a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:48.268110 22309 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:48.269109 22309 tablet_bootstrap.cc:492] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:48.269790 22309 ts_tablet_manager.cc:1403] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.005s
I20251215 08:58:48.272167 22314 tablet_copy_client.cc:806] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:48.272819 22314 tablet_copy_client.cc:670] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:48.272651 22309 raft_consensus.cc:359] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.273543 22309 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:48.274219 22309 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.276576 22309 ts_tablet_manager.cc:1434] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:58:48.278271 22314 tablet_copy_client.cc:538] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:48.278317 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-9d1675f3765f488e9d58a2ba2993457e received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:48.280740 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-9d1675f3765f488e9d58a2ba2993457e on tablet 9d1675f3765f488e9d58a2ba2993457e with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:48.289054 22314 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:48.314149 22314 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:48.315085 22314 tablet_bootstrap.cc:492] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:48.315719 22314 ts_tablet_manager.cc:1403] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.005s
I20251215 08:58:48.317942 22314 raft_consensus.cc:359] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.318511 22314 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:48.319078 22314 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.320989 22314 ts_tablet_manager.cc:1434] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251215 08:58:48.322839 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-3cf7b0165c00477b9c1b94c6a89052a6 received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:48.323202 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-3cf7b0165c00477b9c1b94c6a89052a6 on tablet 3cf7b0165c00477b9c1b94c6a89052a6 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:48.332688 22314 ts_tablet_manager.cc:933] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:48.339138 22314 tablet_copy_client.cc:323] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:48.340705 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 1db12563817b44429a055b7dcdd066c3 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:48.341269 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 1db12563817b44429a055b7dcdd066c3 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-1db12563817b44429a055b7dcdd066c3
I20251215 08:58:48.348407 21276 tablet_copy_source_session.cc:215] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:48.353648 22314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db12563817b44429a055b7dcdd066c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:48.370108 22314 tablet_copy_client.cc:806] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:48.370751 22314 tablet_copy_client.cc:670] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:48.374344 22314 tablet_copy_client.cc:538] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:48.383591 22314 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:48.416167 22314 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:48.416965 22314 tablet_bootstrap.cc:492] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:48.417637 22314 ts_tablet_manager.cc:1403] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.011s
I20251215 08:58:48.419832 22314 raft_consensus.cc:359] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.420553 22314 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:48.421208 22314 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:48.426401 22314 ts_tablet_manager.cc:1434] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251215 08:58:48.427850 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-1db12563817b44429a055b7dcdd066c3 received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:48.428257 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-1db12563817b44429a055b7dcdd066c3 on tablet 1db12563817b44429a055b7dcdd066c3 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:48.606560 21977 raft_consensus.cc:1217] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:48.666261 21977 raft_consensus.cc:1217] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:48.806936 21979 raft_consensus.cc:1217] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20251215 08:58:48.864241 21979 raft_consensus.cc:1217] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:49.048887 22321 raft_consensus.cc:1064] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:49.051398 22321 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.059940 21261 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.061112 21409 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.061293 22321 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.061154 21979 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.062429 22300 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.063706 22255 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.078250 22255 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.080283 21259 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.084532 21979 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.091137 21409 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.101748 21164 catalog_manager.cc:5654] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:49.105613 22320 raft_consensus.cc:1064] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:49.108140 22320 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.122355 21335 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.124359 22291 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.124867 21409 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.119513 21977 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.129616 22280 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.131351 22280 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.175995 22280 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.177595 21977 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.178889 21335 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.190994 21409 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.203954 21335 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.205072 21162 catalog_manager.cc:5654] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.218263 21266 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:49.220149 21977 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:49.221313 22325 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:49.223074 22325 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:49.229929 22255 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.236963 21259 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.243548 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 124c9efc80174ca585aaee7e0f8a12f0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:49.245692 21261 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.255107 21977 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:49.249877 21979 raft_consensus.cc:2955] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.256708 21162 catalog_manager.cc:5654] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:49.256629 22280 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:49.262538 21409 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:49.263720 22280 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:49.266644 22320 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.268016 21977 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.272392 21409 raft_consensus.cc:2955] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.283710 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 9d1675f3765f488e9d58a2ba2993457e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:49.285889 21400 tablet_service.cc:1552] Processing DeleteTablet for tablet 124c9efc80174ca585aaee7e0f8a12f0 with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:49.297410 22337 tablet_replica.cc:333] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:49.297355 21163 catalog_manager.cc:5654] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.298672 22337 raft_consensus.cc:2243] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:49.299726 22337 raft_consensus.cc:2272] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:49.305823 22337 ts_tablet_manager.cc:1916] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:49.311550 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1675f3765f488e9d58a2ba2993457e with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:49.325289 22338 tablet_replica.cc:333] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:49.329497 22338 raft_consensus.cc:2243] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:49.330351 22338 raft_consensus.cc:2272] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:49.338093 22280 raft_consensus.cc:1064] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:49.341073 22338 ts_tablet_manager.cc:1916] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:49.340375 22280 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.349426 22337 ts_tablet_manager.cc:1929] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:49.349891 22337 log.cc:1199] T 124c9efc80174ca585aaee7e0f8a12f0 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/124c9efc80174ca585aaee7e0f8a12f0
I20251215 08:58:49.351356 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet 124c9efc80174ca585aaee7e0f8a12f0 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:49.353376 21413 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.354242 21977 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.354704 22323 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.355718 22320 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.357560 22338 ts_tablet_manager.cc:1929] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:49.357973 22338 log.cc:1199] T 9d1675f3765f488e9d58a2ba2993457e P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/9d1675f3765f488e9d58a2ba2993457e
I20251215 08:58:49.358944 21335 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:49.359643 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 9d1675f3765f488e9d58a2ba2993457e (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:49.362113 22280 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:49.366827 22291 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.368876 21977 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.380411 21413 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.384265 21335 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.389925 21163 catalog_manager.cc:5654] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:49.413930 21259 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.420950 21335 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:49.421033 21979 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:49.422870 22322 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:49.424183 22280 raft_consensus.cc:1064] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:49.426801 22280 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.430370 22320 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:58:49.432319 22323 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.436517 21977 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.450281 21412 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251215 08:58:49.450305 21979 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251215 08:58:49.452692 22344 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20251215 08:58:49.455439 21335 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251215 08:58:49.457336 22291 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.457201 21333 raft_consensus.cc:2955] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.469352 22320 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:49.471433 21163 catalog_manager.cc:5654] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.485917 22320 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.489741 21400 tablet_service.cc:1552] Processing DeleteTablet for tablet 1db12563817b44429a055b7dcdd066c3 with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:49.488767 21335 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.491000 22337 tablet_replica.cc:333] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:49.491780 22337 raft_consensus.cc:2243] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:49.492501 22337 raft_consensus.cc:2272] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:49.497958 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 1db12563817b44429a055b7dcdd066c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:49.513610 22337 ts_tablet_manager.cc:1916] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:49.519743 21412 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.520794 21163 catalog_manager.cc:5654] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.530301 21977 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.550660 22337 ts_tablet_manager.cc:1929] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:49.551211 22337 log.cc:1199] T 1db12563817b44429a055b7dcdd066c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/1db12563817b44429a055b7dcdd066c3
I20251215 08:58:49.553318 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet 1db12563817b44429a055b7dcdd066c3 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:49.558269 21259 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:49.562053 21412 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251215 08:58:49.562732 21977 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251215 08:58:49.566442 22280 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:49.567560 22345 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:49.573680 22345 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.575112 21412 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.579489 21977 raft_consensus.cc:2955] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:49.584499 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 3cf7b0165c00477b9c1b94c6a89052a6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:49.593811 21164 catalog_manager.cc:5654] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 2 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:49.605273 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf7b0165c00477b9c1b94c6a89052a6 with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:49.606156 22338 tablet_replica.cc:333] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:49.607197 22338 raft_consensus.cc:2243] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:49.607898 22338 raft_consensus.cc:2272] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:49.610913 22338 ts_tablet_manager.cc:1916] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:49.625380 22338 ts_tablet_manager.cc:1929] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20251215 08:58:49.625713 22338 log.cc:1199] T 3cf7b0165c00477b9c1b94c6a89052a6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/3cf7b0165c00477b9c1b94c6a89052a6
I20251215 08:58:49.626850 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 3cf7b0165c00477b9c1b94c6a89052a6 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:50.655491 21335 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
W20251215 08:58:50.661082 21304 consensus_peers.cc:597] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac8397b87ffc453b8f468eee77a8e410. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:50.662173 21259 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:50.662102 21412 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:50.663594 22255 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.664356 22351 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.673161 22351 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.674783 21259 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.675324 21412 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.689556 21409 consensus_queue.cc:237] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:50.692070 21163 catalog_manager.cc:5654] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 2 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.705035 21335 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:50.706038 21378 consensus_peers.cc:597] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4e724e065b349faafa05321d2582972. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:50.706583 22228 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.708894 21259 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:50.711853 22228 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.719944 21332 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.718360 22228 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.730536 21261 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.759670 21164 catalog_manager.cc:5654] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.760176 21409 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
W20251215 08:58:50.769255 21378 consensus_peers.cc:597] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 676d761e0efc443cad981187af43b8d9. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:50.769752 21332 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:50.770242 21261 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:50.771425 22349 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.772153 22361 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.780405 22361 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.782286 21261 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.787881 21332 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.794423 21261 consensus_queue.cc:237] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:50.799274 21161 catalog_manager.cc:5654] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
W20251215 08:58:50.805955 21229 consensus_peers.cc:597] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 458cec829de44303951b997be68accde. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:50.806995 21409 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:50.808506 22320 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.812780 21335 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:50.816999 22320 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:50.834933 22280 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.837067 21332 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.843245 21409 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:50.859015 21164 catalog_manager.cc:5654] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:51.106194 22369 ts_tablet_manager.cc:933] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:51.108239 22369 tablet_copy_client.cc:323] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:51.109706 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet a4e724e065b349faafa05321d2582972 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:51.110235 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet a4e724e065b349faafa05321d2582972 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-a4e724e065b349faafa05321d2582972
I20251215 08:58:51.115983 21425 tablet_copy_source_session.cc:215] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:51.118752 22369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e724e065b349faafa05321d2582972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:51.128502 22371 ts_tablet_manager.cc:933] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:51.130422 22371 tablet_copy_client.cc:323] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:51.133431 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet ac8397b87ffc453b8f468eee77a8e410 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:51.133919 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet ac8397b87ffc453b8f468eee77a8e410 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-ac8397b87ffc453b8f468eee77a8e410
I20251215 08:58:51.141263 22369 tablet_copy_client.cc:806] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:51.141613 21351 tablet_copy_source_session.cc:215] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:51.141878 22369 tablet_copy_client.cc:670] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:51.145501 22369 tablet_copy_client.cc:538] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:51.145754 22371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac8397b87ffc453b8f468eee77a8e410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:51.153283 22369 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:51.160128 22371 tablet_copy_client.cc:806] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:51.160655 22371 tablet_copy_client.cc:670] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:51.165189 22371 tablet_copy_client.cc:538] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:51.175606 22371 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:51.183668 22369 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:51.184562 22369 tablet_bootstrap.cc:492] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:51.185278 22369 ts_tablet_manager.cc:1403] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.000s
I20251215 08:58:51.188486 22369 raft_consensus.cc:359] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.189311 22369 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:51.189958 22369 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.192705 22369 ts_tablet_manager.cc:1434] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251215 08:58:51.194186 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-a4e724e065b349faafa05321d2582972 received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:51.194615 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-a4e724e065b349faafa05321d2582972 on tablet a4e724e065b349faafa05321d2582972 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:51.199617 22371 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:51.200564 22371 tablet_bootstrap.cc:492] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:51.201161 22371 ts_tablet_manager.cc:1403] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.008s
I20251215 08:58:51.203775 22371 raft_consensus.cc:359] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.204406 22371 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:51.205058 22371 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.207680 22371 ts_tablet_manager.cc:1434] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251215 08:58:51.209116 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-ac8397b87ffc453b8f468eee77a8e410 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:51.209504 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-ac8397b87ffc453b8f468eee77a8e410 on tablet ac8397b87ffc453b8f468eee77a8e410 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:51.285907 22371 ts_tablet_manager.cc:933] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:51.287498 22371 tablet_copy_client.cc:323] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:51.288760 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 458cec829de44303951b997be68accde from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:51.289305 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 458cec829de44303951b997be68accde from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-458cec829de44303951b997be68accde
I20251215 08:58:51.296053 21276 tablet_copy_source_session.cc:215] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:51.299468 22371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 458cec829de44303951b997be68accde. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:51.312849 22369 ts_tablet_manager.cc:933] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:51.314280 22371 tablet_copy_client.cc:806] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:51.315838 22371 tablet_copy_client.cc:670] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:51.317842 22369 tablet_copy_client.cc:323] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.67:37447
I20251215 08:58:51.318930 22371 tablet_copy_client.cc:538] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:51.319267 21425 tablet_copy_service.cc:140] P 195a56ac72a84018880bfb3bf6e6ef1d: Received BeginTabletCopySession request for tablet 676d761e0efc443cad981187af43b8d9 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:59102)
I20251215 08:58:51.319712 21425 tablet_copy_service.cc:161] P 195a56ac72a84018880bfb3bf6e6ef1d: Beginning new tablet copy session on tablet 676d761e0efc443cad981187af43b8d9 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:59102: session id = f0824fa5450345719cac22fc6ae3ce2e-676d761e0efc443cad981187af43b8d9
I20251215 08:58:51.326371 22371 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:51.329617 21425 tablet_copy_source_session.cc:215] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:51.333276 22369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676d761e0efc443cad981187af43b8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:51.350068 22371 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:51.351132 22371 tablet_bootstrap.cc:492] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:51.351833 22371 ts_tablet_manager.cc:1403] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.000s
I20251215 08:58:51.352087 22369 tablet_copy_client.cc:806] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:51.352670 22369 tablet_copy_client.cc:670] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:51.354848 22371 raft_consensus.cc:359] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.355626 22371 raft_consensus.cc:740] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:51.356236 22371 consensus_queue.cc:260] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.358057 22371 ts_tablet_manager.cc:1434] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20251215 08:58:51.361253 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-458cec829de44303951b997be68accde received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:51.361632 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-458cec829de44303951b997be68accde on tablet 458cec829de44303951b997be68accde with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:51.361658 22369 tablet_copy_client.cc:538] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:51.374251 22369 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:51.391902 22369 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:51.392779 22369 tablet_bootstrap.cc:492] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:51.393599 22369 ts_tablet_manager.cc:1403] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20251215 08:58:51.396392 22369 raft_consensus.cc:359] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.397166 22369 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:51.397791 22369 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:51.399701 22369 ts_tablet_manager.cc:1434] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251215 08:58:51.402469 21425 tablet_copy_service.cc:342] P 195a56ac72a84018880bfb3bf6e6ef1d: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-676d761e0efc443cad981187af43b8d9 received from {username='slave'} at 127.0.0.1:59102
I20251215 08:58:51.402834 21425 tablet_copy_service.cc:434] P 195a56ac72a84018880bfb3bf6e6ef1d: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-676d761e0efc443cad981187af43b8d9 on tablet 676d761e0efc443cad981187af43b8d9 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:51.552934 21979 raft_consensus.cc:1217] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:51.567756 21979 raft_consensus.cc:1217] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20251215 08:58:51.765362 21979 raft_consensus.cc:1217] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:51.856276 21979 raft_consensus.cc:1217] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:52.008968 22361 raft_consensus.cc:1064] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:52.011274 22361 consensus_queue.cc:237] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:52.018234 21335 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.021080 21979 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.019593 22382 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20251215 08:58:52.022606 22228 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.027369 21259 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.028800 22382 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.043597 22382 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.044559 21333 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.055260 21979 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.114224 21161 catalog_manager.cc:5654] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:52.179618 21261 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.215386 22255 raft_consensus.cc:1064] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:52.220117 22255 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:52.245249 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 47564.5us
I20251215 08:58:52.269146 21979 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251215 08:58:52.275218 22395 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.277856 22382 raft_consensus.cc:1064] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:52.281864 22382 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:52.298447 21409 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251215 08:58:52.307318 21979 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.309012 22325 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.308305 21259 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.309970 21266 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251215 08:58:52.311079 22228 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:52.312853 22361 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.321210 21333 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.322645 22382 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:52.329731 22395 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.336448 21412 consensus_queue.cc:237] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:52.350906 22361 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.352630 21977 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.375149 21979 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:52.379664 22382 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:52.409669 21333 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.411830 21261 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:52.453500 21161 catalog_manager.cc:5654] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.455183 22382 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:52.466765 22392 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.491925 21261 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.493558 21977 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.497304 22399 raft_consensus.cc:1064] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:52.504113 22399 consensus_queue.cc:237] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:52.507463 22413 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.519547 21258 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.521263 21415 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.543402 21162 catalog_manager.cc:5654] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.554076 21977 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:58:52.567260 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 51130.2us
I20251215 08:58:52.570457 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet a4e724e065b349faafa05321d2582972 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:52.570874 21415 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:52.572599 21332 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:52.580019 21978 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251215 08:58:52.581900 21979 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:52.584174 22394 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:58:52.584710 21332 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:52.587028 21409 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251215 08:58:52.569375 21261 raft_consensus.cc:2955] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.591148 22392 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.592615 21979 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.595237 22415 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20251215 08:58:52.596081 22255 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:52.596158 22413 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:58:52.598613 21980 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.600602 21409 raft_consensus.cc:2955] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.602380 21413 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.604342 22380 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.608172 22383 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:52.610338 21332 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:52.612242 22228 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.624024 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 676d761e0efc443cad981187af43b8d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:52.625279 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet ac8397b87ffc453b8f468eee77a8e410 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:52.625558 22380 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:52.620244 21164 catalog_manager.cc:5654] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.632920 21336 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.646663 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e724e065b349faafa05321d2582972 with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:58:52.650875 21979 raft_consensus.cc:2955] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.657377 21164 catalog_manager.cc:5654] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 2 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.671840 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet ac8397b87ffc453b8f468eee77a8e410 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:52.692879 21161 catalog_manager.cc:5654] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:52.707386 22398 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.718874 22420 tablet_replica.cc:333] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:52.720155 22420 raft_consensus.cc:2243] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:52.720944 22420 raft_consensus.cc:2272] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:52.725050 21339 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.729522 22418 tablet_replica.cc:333] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:58:52.730715 22418 raft_consensus.cc:2243] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:52.731544 22418 raft_consensus.cc:2272] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:52.734015 21415 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.742594 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet 676d761e0efc443cad981187af43b8d9 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:52.744508 22418 ts_tablet_manager.cc:1916] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:52.748560 21162 catalog_manager.cc:5654] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:52.759311 21978 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.771486 22418 ts_tablet_manager.cc:1929] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:52.772056 22418 log.cc:1199] T a4e724e065b349faafa05321d2582972 P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/a4e724e065b349faafa05321d2582972
I20251215 08:58:52.779280 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet a4e724e065b349faafa05321d2582972 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:52.781838 22420 ts_tablet_manager.cc:1916] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:52.799858 21258 consensus_queue.cc:237] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:52.804553 21978 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:52.805347 22420 ts_tablet_manager.cc:1929] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20251215 08:58:52.805863 22420 log.cc:1199] T ac8397b87ffc453b8f468eee77a8e410 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/ac8397b87ffc453b8f468eee77a8e410
I20251215 08:58:52.806466 21336 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:52.806375 22398 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:52.807808 22420 tablet_replica.cc:333] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:52.808125 22383 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:52.808984 22420 raft_consensus.cc:2243] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:52.809731 22420 raft_consensus.cc:2272] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:52.809778 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet ac8397b87ffc453b8f468eee77a8e410 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:52.815923 22380 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.817986 21978 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.818385 21336 raft_consensus.cc:2955] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.820876 22420 ts_tablet_manager.cc:1916] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:52.831509 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 458cec829de44303951b997be68accde with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:52.833259 21161 catalog_manager.cc:5654] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:52.838138 22420 ts_tablet_manager.cc:1929] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:52.838686 22420 log.cc:1199] T 676d761e0efc443cad981187af43b8d9 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/676d761e0efc443cad981187af43b8d9
I20251215 08:58:52.846995 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet 676d761e0efc443cad981187af43b8d9 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:52.849256 21400 tablet_service.cc:1552] Processing DeleteTablet for tablet 458cec829de44303951b997be68accde with delete_type TABLET_DATA_TOMBSTONED (TS 195a56ac72a84018880bfb3bf6e6ef1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32792
I20251215 08:58:52.851967 22421 tablet_replica.cc:333] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: stopping tablet replica
I20251215 08:58:52.859544 22421 raft_consensus.cc:2243] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:52.860364 22421 raft_consensus.cc:2272] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:52.863286 22421 ts_tablet_manager.cc:1916] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:52.874514 22421 ts_tablet_manager.cc:1929] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:58:52.874851 22421 log.cc:1199] T 458cec829de44303951b997be68accde P 195a56ac72a84018880bfb3bf6e6ef1d: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-2-root/wals/458cec829de44303951b997be68accde
I20251215 08:58:52.876235 21151 catalog_manager.cc:4985] TS 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): tablet 458cec829de44303951b997be68accde (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:58:53.908977 21261 consensus_queue.cc:237] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
W20251215 08:58:53.915490 21229 consensus_peers.cc:597] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f1c639a483f4827997eae6f3a6e375d. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:53.915699 21415 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:53.916649 21336 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:53.917084 22398 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:53.917928 22423 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:53.924881 22380 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:53.926390 21415 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:53.933864 21339 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:53.941919 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:53.944164 21162 catalog_manager.cc:5654] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:53.951872 21336 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:53.984647 21409 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:53.987787 22392 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:53.990280 21261 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:53.994102 22393 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251215 08:58:53.999194 21304 consensus_peers.cc:597] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3680af750ef47df88bf5be33eb18fc7. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:54.017900 22393 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.026919 21261 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.040303 21336 consensus_queue.cc:237] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.040122 21409 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.046083 21415 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251215 08:58:54.047204 21304 consensus_peers.cc:597] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d93e1c6de78493c926c5be7096cfb31. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:54.047775 22429 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:54.059126 21261 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:54.065317 22433 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:54.060837 21164 catalog_manager.cc:5654] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.085316 22393 raft_consensus.cc:2955] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.098462 21415 raft_consensus.cc:2955] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.108124 21161 catalog_manager.cc:5654] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251215 08:58:54.107949 21336 consensus_queue.cc:237] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.109622 21261 raft_consensus.cc:2955] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.119607 21415 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:54.120695 22433 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:58:54.121799 21266 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 08:58:54.124603 22393 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:54.131322 21336 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "15d0087ada21421eacebc646b271e145"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:59320
I20251215 08:58:54.132056 21336 raft_consensus.cc:606] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:58:54.136390 21304 consensus_peers.cc:597] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 -> Peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831): Couldn't send request to peer f0824fa5450345719cac22fc6ae3ce2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7edcdb5749ae4e59ab17648fe97390e2. This is attempt 1: this message will repeat every 5th retry.
I20251215 08:58:54.141206 22429 raft_consensus.cc:993] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.141680 22429 raft_consensus.cc:1081] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.147655 21979 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "15d0087ada21421eacebc646b271e145"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46170
I20251215 08:58:54.148555 21979 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:54.149068 21979 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.152320 21409 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.154466 22429 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.157298 21979 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.159668 21979 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:54.160851 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:54.161518 21415 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.162281 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:54.162818 21339 raft_consensus.cc:3055] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:54.163086 21339 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:54.163862 21339 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.166002 21339 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.170056 21415 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.166416 21266 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.172556 21339 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.176909 21946 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:54.190313 21164 catalog_manager.cc:5654] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index -1 to 2, NON_VOTER f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) added. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } } }
I20251215 08:58:54.200621 22445 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:54.219472 22445 raft_consensus.cc:697] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:58:54.219472 21266 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:58:54.220180 21266 raft_consensus.cc:606] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.220482 22445 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.233263 21336 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:59320
I20251215 08:58:54.233793 21336 raft_consensus.cc:606] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.236351 21341 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
mode: GRACEFUL
new_leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:59320
I20251215 08:58:54.236910 21341 raft_consensus.cc:606] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Received request to transfer leadership to TS 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:54.255115 21409 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "676d761e0efc443cad981187af43b8d9"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:58:54.255579 21409 raft_consensus.cc:606] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.258407 21339 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c61929339a494e3fb5f327784d94d817"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:59320
I20251215 08:58:54.258945 21339 raft_consensus.cc:606] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.260730 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251215 08:58:54.260087 21164 catalog_manager.cc:5654] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:54.262121 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:54.263034 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:54.276949 22392 raft_consensus.cc:993] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: : Instructing follower 195a56ac72a84018880bfb3bf6e6ef1d to start an election
I20251215 08:58:54.277616 22393 raft_consensus.cc:1081] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Signalling peer 195a56ac72a84018880bfb3bf6e6ef1d to start an election
I20251215 08:58:54.279013 21409 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:32802
I20251215 08:58:54.279518 21409 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:54.279827 21409 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.286186 21409 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:54.288874 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:54.289561 21266 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.289597 21409 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:54.289809 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:54.290349 21339 raft_consensus.cc:3055] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:54.290635 21339 raft_consensus.cc:740] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:54.291273 21339 consensus_queue.cc:260] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:54.292335 21339 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.296307 21266 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:54.298563 21339 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:54.302775 21381 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:54.304064 22426 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:54.304502 22426 raft_consensus.cc:697] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:54.305347 22426 consensus_queue.cc:237] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:54.316330 22415 raft_consensus.cc:993] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.316771 22415 raft_consensus.cc:1081] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.318367 21979 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "676d761e0efc443cad981187af43b8d9"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46158
I20251215 08:58:54.318873 21979 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:54.319200 21979 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.321069 21164 catalog_manager.cc:5654] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:58:54.327245 21979 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.329766 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:54.330300 21341 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.330854 22393 raft_consensus.cc:993] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.331310 22429 raft_consensus.cc:1081] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.332831 21978 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46170
I20251215 08:58:54.333357 21978 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:54.333701 21978 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.334360 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:54.334888 21409 raft_consensus.cc:3055] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:54.335140 21409 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:54.335811 21409 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.336791 21409 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.337136 21979 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:54.336719 21341 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.339031 21978 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.339496 21944 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:54.340332 22445 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:54.340791 22445 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:58:54.341601 22445 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.342700 21978 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:54.342772 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:54.343544 21341 raft_consensus.cc:3055] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:54.343847 21341 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:54.343889 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:54.344519 21412 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.344673 21341 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.346560 21341 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.346465 21409 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.354619 21341 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.355800 21944 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:54.355671 21164 catalog_manager.cc:5654] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:54.357488 22445 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:54.358654 22445 raft_consensus.cc:697] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:58:54.359541 22445 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.361932 21412 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.375342 21164 catalog_manager.cc:5654] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:54.454133 22452 ts_tablet_manager.cc:933] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193)
I20251215 08:58:54.456511 22452 tablet_copy_client.cc:323] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.65:46193
I20251215 08:58:54.458308 21276 tablet_copy_service.cc:140] P beefe15185ac42ef9a20251007f39a56: Received BeginTabletCopySession request for tablet 4f1c639a483f4827997eae6f3a6e375d from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:35486)
I20251215 08:58:54.458833 21276 tablet_copy_service.cc:161] P beefe15185ac42ef9a20251007f39a56: Beginning new tablet copy session on tablet 4f1c639a483f4827997eae6f3a6e375d from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:35486: session id = f0824fa5450345719cac22fc6ae3ce2e-4f1c639a483f4827997eae6f3a6e375d
I20251215 08:58:54.464778 21276 tablet_copy_source_session.cc:215] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:54.467422 22452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1c639a483f4827997eae6f3a6e375d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:54.484710 22452 tablet_copy_client.cc:806] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:54.485370 22452 tablet_copy_client.cc:670] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:54.489398 22452 tablet_copy_client.cc:538] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:54.497401 22452 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:54.501529 22398 raft_consensus.cc:993] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.502112 22380 raft_consensus.cc:1081] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.503511 21979 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46154
I20251215 08:58:54.503916 21979 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:54.504205 21979 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:54.512708 21979 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.518968 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:54.520347 21266 raft_consensus.cc:3055] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:58:54.520763 21266 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:54.521622 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:54.522195 21412 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:54.522271 21979 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:54.521647 21266 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.531250 21266 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:54.529155 21412 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:58:54.538540 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:54.538970 22456 ts_tablet_manager.cc:933] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:54.539474 22445 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:58:54.539985 22445 raft_consensus.cc:697] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:58:54.540868 22445 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.555927 22456 tablet_copy_client.cc:323] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:54.556777 21266 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:58:54.564546 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet 7edcdb5749ae4e59ab17648fe97390e2 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:54.565140 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet 7edcdb5749ae4e59ab17648fe97390e2 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-7edcdb5749ae4e59ab17648fe97390e2
I20251215 08:58:54.520359 22455 ts_tablet_manager.cc:933] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:54.564754 21164 catalog_manager.cc:5654] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 2 to 3, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 3 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:54.568126 22452 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:54.569083 22452 tablet_bootstrap.cc:492] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:54.569780 22452 ts_tablet_manager.cc:1403] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.073s user 0.018s sys 0.011s
I20251215 08:58:54.572244 22460 ts_tablet_manager.cc:933] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Initiating tablet copy from peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:54.579376 21351 tablet_copy_source_session.cc:215] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:54.579757 22455 tablet_copy_client.cc:323] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:54.572896 22452 raft_consensus.cc:359] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.582960 22452 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:54.583021 21350 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet 5d93e1c6de78493c926c5be7096cfb31 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:54.583645 21350 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet 5d93e1c6de78493c926c5be7096cfb31 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-5d93e1c6de78493c926c5be7096cfb31
I20251215 08:58:54.583719 22452 consensus_queue.cc:260] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.587419 22452 ts_tablet_manager.cc:1434] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20251215 08:58:54.589278 21276 tablet_copy_service.cc:342] P beefe15185ac42ef9a20251007f39a56: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-4f1c639a483f4827997eae6f3a6e375d received from {username='slave'} at 127.0.0.1:35486
I20251215 08:58:54.589708 21276 tablet_copy_service.cc:434] P beefe15185ac42ef9a20251007f39a56: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-4f1c639a483f4827997eae6f3a6e375d on tablet 4f1c639a483f4827997eae6f3a6e375d with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.593168 21350 tablet_copy_source_session.cc:215] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:54.596323 22455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d93e1c6de78493c926c5be7096cfb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:54.604707 22456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edcdb5749ae4e59ab17648fe97390e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:54.616252 22460 tablet_copy_client.cc:323] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.163.66:36421
I20251215 08:58:54.620929 21351 tablet_copy_service.cc:140] P 0a2fcb89589c401d8efc8d86da4ba0f4: Received BeginTabletCopySession request for tablet b3680af750ef47df88bf5be33eb18fc7 from peer f0824fa5450345719cac22fc6ae3ce2e ({username='slave'} at 127.0.0.1:47830)
I20251215 08:58:54.621486 21351 tablet_copy_service.cc:161] P 0a2fcb89589c401d8efc8d86da4ba0f4: Beginning new tablet copy session on tablet b3680af750ef47df88bf5be33eb18fc7 from peer f0824fa5450345719cac22fc6ae3ce2e at {username='slave'} at 127.0.0.1:47830: session id = f0824fa5450345719cac22fc6ae3ce2e-b3680af750ef47df88bf5be33eb18fc7
I20251215 08:58:54.624909 22455 tablet_copy_client.cc:806] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:54.625537 22455 tablet_copy_client.cc:670] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:54.627106 22456 tablet_copy_client.cc:806] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:54.627671 22456 tablet_copy_client.cc:670] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:54.633399 22455 tablet_copy_client.cc:538] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:54.632198 22456 tablet_copy_client.cc:538] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:54.640878 21351 tablet_copy_source_session.cc:215] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 08:58:54.642592 22455 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:54.643821 22460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3680af750ef47df88bf5be33eb18fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 08:58:54.659582 22456 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:54.665096 22460 tablet_copy_client.cc:806] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 0 data blocks...
I20251215 08:58:54.665685 22460 tablet_copy_client.cc:670] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Starting download of 1 WAL segments...
I20251215 08:58:54.668334 22455 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:54.669211 22455 tablet_bootstrap.cc:492] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:54.669854 22455 ts_tablet_manager.cc:1403] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.006s
I20251215 08:58:54.679723 22455 raft_consensus.cc:359] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.681869 22455 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:54.683007 22392 raft_consensus.cc:993] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.683468 22392 raft_consensus.cc:1081] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:54.684144 22460 tablet_copy_client.cc:538] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 08:58:54.685709 21979 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c61929339a494e3fb5f327784d94d817"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46170
I20251215 08:58:54.686220 21979 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:54.686549 21979 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.688779 22456 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:54.689630 22456 tablet_bootstrap.cc:492] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:54.690251 22456 ts_tablet_manager.cc:1403] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.007s
I20251215 08:58:54.692970 22456 raft_consensus.cc:359] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.693814 22456 raft_consensus.cc:740] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:54.694504 22460 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap starting.
I20251215 08:58:54.693981 22455 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.694439 22456 consensus_queue.cc:260] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.700698 22456 ts_tablet_manager.cc:1434] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.010s user 0.003s sys 0.005s
I20251215 08:58:54.701220 22455 ts_tablet_manager.cc:1434] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20251215 08:58:54.702299 21351 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-7edcdb5749ae4e59ab17648fe97390e2 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:54.702746 21351 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-7edcdb5749ae4e59ab17648fe97390e2 on tablet 7edcdb5749ae4e59ab17648fe97390e2 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.703383 21349 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-5d93e1c6de78493c926c5be7096cfb31 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:54.703768 21349 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-5d93e1c6de78493c926c5be7096cfb31 on tablet 5d93e1c6de78493c926c5be7096cfb31 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.692401 21979 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.720081 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:54.720300 21979 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:54.720697 21266 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.723415 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:54.723902 21336 raft_consensus.cc:3055] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:54.724166 21336 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:54.724931 21336 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.728539 21336 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:54.737746 21341 raft_consensus.cc:1275] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:54.731909 22460 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 08:58:54.739228 22228 consensus_queue.cc:1048] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:54.740005 22460 tablet_bootstrap.cc:492] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Bootstrap complete.
I20251215 08:58:54.728617 21266 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.741068 21946 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:54.741335 22460 ts_tablet_manager.cc:1403] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.018s
I20251215 08:58:54.741873 22445 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:54.742343 22445 raft_consensus.cc:697] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:58:54.743191 22445 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:54.744087 22460 raft_consensus.cc:359] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.744761 21336 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:54.744935 22460 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Initialized, Role: LEARNER
I20251215 08:58:54.746143 22460 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: NON_VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: true } }
I20251215 08:58:54.749397 22460 ts_tablet_manager.cc:1434] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251215 08:58:54.751695 21349 tablet_copy_service.cc:342] P 0a2fcb89589c401d8efc8d86da4ba0f4: Request end of tablet copy session f0824fa5450345719cac22fc6ae3ce2e-b3680af750ef47df88bf5be33eb18fc7 received from {username='slave'} at 127.0.0.1:47830
I20251215 08:58:54.752121 21349 tablet_copy_service.cc:434] P 0a2fcb89589c401d8efc8d86da4ba0f4: ending tablet copy session f0824fa5450345719cac22fc6ae3ce2e-b3680af750ef47df88bf5be33eb18fc7 on tablet b3680af750ef47df88bf5be33eb18fc7 with peer f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:54.764021 21164 catalog_manager.cc:5654] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:54.769595 21266 raft_consensus.cc:1275] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:54.771675 22228 consensus_queue.cc:1048] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:54.790889 21412 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:54.793447 22446 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:58:54.794479 21341 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:54.795908 22445 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:54.845059 21339 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:54.849434 22445 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:54.874930 21412 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:54.878906 22475 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20251215 08:58:54.948770 21412 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:54.951709 22479 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:58:54.957154 21336 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:54.962288 22445 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:54.977926 21412 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:58:54.980947 22479 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:58:54.996635 21266 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:58:55.004528 22479 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:55.010874 21979 raft_consensus.cc:1217] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:55.029670 21979 raft_consensus.cc:1217] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:55.078779 21979 raft_consensus.cc:1217] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:55.103816 21978 raft_consensus.cc:1217] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251215 08:58:55.155417 21266 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:55.156646 22479 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:55.173481 21336 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:55.174763 22475 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:55.323225 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:55.368721 21261 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:58:55.369313 21261 raft_consensus.cc:606] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Received request to transfer leadership to TS 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:55.371682 21261 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:58:55.373142 21261 raft_consensus.cc:606] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:55.375622 21409 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:58:55.376139 21409 raft_consensus.cc:606] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:55.381492 21409 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:58:55.382004 21409 raft_consensus.cc:606] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:55.385324 21409 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:58:55.385879 21409 raft_consensus.cc:606] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:55.386785 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:58:55.388322 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:58:55.388914 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:55.501425 22392 raft_consensus.cc:1064] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:55.503917 22392 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.511019 21979 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.511751 21266 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.513185 21409 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.513468 22433 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.514302 22393 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.515275 22429 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.538856 22429 raft_consensus.cc:1064] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:55.543850 22429 consensus_queue.cc:237] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.550450 22433 raft_consensus.cc:1064] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:55.552320 21412 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.552572 22392 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.553568 22433 consensus_queue.cc:237] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.553658 21412 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.558696 22393 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.591737 21978 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.602310 21979 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.604144 22497 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.617136 22487 raft_consensus.cc:993] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:55.617676 22487 raft_consensus.cc:1081] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:55.619269 21979 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46158
I20251215 08:58:55.619911 21979 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:55.620328 21979 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:55.620003 21164 catalog_manager.cc:5654] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.625510 22473 raft_consensus.cc:1064] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: attempting to promote NON_VOTER f0824fa5450345719cac22fc6ae3ce2e to VOTER
I20251215 08:58:55.627965 22473 consensus_queue.cc:237] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.633080 21979 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.641180 21977 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.642704 22496 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.646996 21415 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.649240 22398 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.654696 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:55.655599 21266 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:55.657490 21977 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 LEARNER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.661970 22489 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251215 08:58:55.686045 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 132413us
I20251215 08:58:55.689453 21415 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.692891 21261 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.695123 21979 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:55.695992 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:55.696745 21409 raft_consensus.cc:3055] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:55.697324 21409 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:55.697778 21261 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.698309 21409 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.699608 21409 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:55.700819 21266 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:55.701869 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:55.702584 22476 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:55.702999 22476 raft_consensus.cc:697] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:58:55.703754 22476 consensus_queue.cc:237] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.709493 21336 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251215 08:58:55.715813 22499 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.715925 22393 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.716905 22429 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:58:55.724895 21409 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:55.728329 21336 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:55.730437 21161 catalog_manager.cc:5654] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:58:55.733546 22487 raft_consensus.cc:993] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:58:55.734042 22487 raft_consensus.cc:1081] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:58:55.734707 22489 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:58:55.737725 21339 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:58:55.738291 21339 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:55.738624 21339 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:55.741242 21979 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:55.741138 22398 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.742555 22499 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:55.742172 21412 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.754268 21266 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.755848 22497 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.763273 21979 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.786556 21412 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:58:55.789472 21339 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:55.801172 21339 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:55.819881 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:55.820765 21412 raft_consensus.cc:3055] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:55.821136 21412 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:55.830493 21412 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:55.835740 21412 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:55.844799 21161 catalog_manager.cc:5654] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.824923 22499 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20251215 08:58:55.849493 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:55.851435 21412 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:55.852460 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:55.855285 21258 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:55.877090 21412 raft_consensus.cc:2955] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:55.874915 22429 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:55.888747 21258 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:55.995553 22516 raft_consensus.cc:993] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: : Instructing follower 195a56ac72a84018880bfb3bf6e6ef1d to start an election
I20251215 08:58:56.011512 22510 raft_consensus.cc:1081] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Signalling peer 195a56ac72a84018880bfb3bf6e6ef1d to start an election
I20251215 08:58:56.003095 22487 raft_consensus.cc:993] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:58:56.016872 21163 catalog_manager.cc:5654] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 3 to 4, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:56.039770 22487 raft_consensus.cc:1081] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:58:56.057335 22494 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:56.073487 21149 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet b3680af750ef47df88bf5be33eb18fc7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:58:56.063501 21413 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:58:56.086455 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 117765us
W20251215 08:58:56.093273 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 129042us
I20251215 08:58:56.094641 21979 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:56.095254 22496 raft_consensus.cc:2955] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:56.102260 21332 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:58:56.102905 21332 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:56.103228 21332 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:56.104279 22429 raft_consensus.cc:697] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:56.105340 22429 consensus_queue.cc:237] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:56.107923 21976 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:56.133719 21412 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:32818
I20251215 08:58:56.134363 21412 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:56.134735 21412 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:58:56.138206 22521 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.076s user 0.000s sys 0.004s
I20251215 08:58:56.137715 21409 raft_consensus.cc:2955] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:57.396862 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1122) took 1600 ms. Trace:
I20251215 08:58:57.397379 21259 rpcz_store.cc:276] 1215 08:58:55.796765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:55.797076 (+ 311us) service_pool.cc:225] Handling call
1215 08:58:55.797809 (+ 733us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:58:55.824114 (+ 26305us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:58:55.824136 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:58:55.827530 (+ 3394us) log.cc:844] Serialized 564 byte log entry
1215 08:58:56.063187 (+235657us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:58:56.063237 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:56.063325 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:58:56.121692 (+ 58367us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:58:57.396461 (+1274769us) raft_consensus.cc:1685] finished
1215 08:58:57.396487 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:58:57.396763 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8904448,"thread_start_us":235376,"threads_started":1,"wal-append.queue_time_us":233545}
W20251215 08:58:57.417759 21413 consensus_meta.cc:287] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 1.339s user 0.010s sys 0.000s
I20251215 08:58:57.418848 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1147) took 1356 ms. Trace:
I20251215 08:58:57.419320 21413 rpcz_store.cc:276] 1215 08:58:56.062680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:56.062819 (+ 139us) service_pool.cc:225] Handling call
1215 08:58:56.063147 (+ 328us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:56.063374 (+ 227us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:58:57.418395 (+1355021us) log.cc:844] Serialized 14 byte log entry
1215 08:58:57.418540 (+ 145us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:58:57.418575 (+ 35us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:58:57.418602 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:57.418647 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:58:57.418764 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:58:57.434939 22535 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.435077 22534 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.436218 22534 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.439504 22534 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.440068 22534 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.440622 22534 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.443432 22534 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.435737 22535 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.446015 22535 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.483834 22519 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.484352 22519 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.486153 22519 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:57.492877 21195 debug-util.cc:398] Leaking SignalData structure 0x7b0800091420 after lost signal to thread 20112
I20251215 08:58:57.507421 22534 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.508049 22534 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.510023 22534 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.510558 22534 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.511034 22534 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.511934 22518 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.512538 22518 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.512878 22534 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.514950 22518 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.515764 22519 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.516247 22519 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.518330 22519 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.519066 22519 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.519321 22426 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.519591 22518 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.519553 22519 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.519869 22426 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:57.520154 22518 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.521502 22518 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.521559 22426 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.521953 22518 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.521518 22519 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:57.519434 21230 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.686s user 0.078s sys 0.043s
I20251215 08:58:57.522420 22518 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:57.523000 21230 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.690s user 0.079s sys 0.043s
W20251215 08:58:57.523413 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 1.6901e+06us
I20251215 08:58:57.524336 22518 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.527544 22537 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.528143 22537 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.528554 22538 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.529371 22538 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.530365 22537 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.531615 22537 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.532226 22538 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.532119 22537 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.533334 22538 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.527593 21161 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36424 (request call id 222) took 1742 ms. Trace:
I20251215 08:58:57.533926 22538 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.534502 22537 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.536206 22538 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.534215 21161 rpcz_store.cc:276] 1215 08:58:55.785644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:55.785807 (+ 163us) service_pool.cc:225] Handling call
1215 08:58:57.527509 (+1741702us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:58:55.875681 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:58:55.876228 (+ 547us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:58:55.876263 (+ 35us) write_op.cc:435] Acquired schema lock
1215 08:58:55.876303 (+ 40us) tablet.cc:660] Decoding operations
1215 08:58:55.876642 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:58:55.876700 (+ 58us) write_op.cc:641] Partition lock acquired for write op
1215 08:58:55.876737 (+ 37us) tablet.cc:683] Acquiring locks for 1 operations
1215 08:58:55.876943 (+ 206us) tablet.cc:699] Row locks acquired
1215 08:58:55.876966 (+ 23us) write_op.cc:260] PREPARE: finished
1215 08:58:55.877145 (+ 179us) write_op.cc:270] Start()
1215 08:58:55.877258 (+ 113us) write_op.cc:276] Timestamp: P: 1765789135877097 usec, L: 0
1215 08:58:55.877277 (+ 19us) op_driver.cc:348] REPLICATION: starting
1215 08:58:55.954030 (+ 76753us) log.cc:844] Serialized 599 byte log entry
1215 08:58:55.999951 (+ 45921us) op_driver.cc:464] REPLICATION: finished
1215 08:58:56.008953 (+ 9002us) write_op.cc:301] APPLY: starting
1215 08:58:56.010992 (+ 2039us) tablet.cc:1365] starting BulkCheckPresence
1215 08:58:56.018473 (+ 7481us) tablet.cc:1368] finished BulkCheckPresence
1215 08:58:56.018584 (+ 111us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:58:56.018854 (+ 270us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:58:56.019065 (+ 211us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1215 08:58:56.019135 (+ 70us) write_op.cc:312] APPLY: finished
1215 08:58:56.019372 (+ 237us) log.cc:844] Serialized 23 byte log entry
1215 08:58:56.019596 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:58:56.019704 (+ 108us) write_op.cc:454] Released schema lock
1215 08:58:56.019756 (+ 52us) write_op.cc:341] FINISH: Updating metrics
1215 08:58:56.040921 (+ 21165us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:58:56.040967 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1445,"apply.run_cpu_time_us":2134,"apply.run_wall_time_us":14273,"num_ops":1,"prepare.queue_time_us":1244,"prepare.run_cpu_time_us":2863,"prepare.run_wall_time_us":87000,"replication_time_us":118403}]]}
I20251215 08:58:57.537916 22518 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.538358 22518 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.539803 22539 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.540347 22539 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.541343 22518 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.541721 22538 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.542271 22537 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.545174 22538 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.545677 22537 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.547252 22538 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.542379 22539 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.547788 22538 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.548269 22538 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.547297 22537 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.549244 22537 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.549603 22537 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.550211 22538 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.551086 22537 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:58:57.557686 21332 consensus_meta.cc:287] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 1.454s user 0.016s sys 0.004s
I20251215 08:58:57.558082 21332 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.559825 21332 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.560266 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:49734 (request call id 1086) took 1458 ms. Trace:
I20251215 08:58:57.560590 21332 rpcz_store.cc:276] 1215 08:58:56.102023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:56.102138 (+ 115us) service_pool.cc:225] Handling call
1215 08:58:57.560177 (+1458039us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:58:57.561579 22518 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.561990 22518 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.563489 22518 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.564802 22518 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.565271 22518 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:57.568480 22429 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.430s user 0.000s sys 0.020s
W20251215 08:58:57.568827 22429 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.430s user 0.000s sys 0.020s
I20251215 08:58:57.569666 22518 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.576890 22541 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.577829 22429 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.577546 22541 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.578387 22429 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.579768 22541 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.580130 22429 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.585328 22494 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.587954 22540 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.589811 22494 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.589993 22540 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.591953 22494 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.592456 22540 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.593048 22540 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.593577 22540 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.595908 22540 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.596437 22540 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.596570 22543 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.596992 22540 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.597362 22543 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.599081 22544 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.601032 22544 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.602188 22494 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.603286 22544 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.602910 22494 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.600551 22540 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.604090 22540 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.604586 22540 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.605527 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1021) took 1732 ms. Trace:
I20251215 08:58:57.605908 21339 rpcz_store.cc:276] 1215 08:58:55.873605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:55.873920 (+ 315us) service_pool.cc:225] Handling call
1215 08:58:57.604880 (+1730960us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b5400164254. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398cb6d58e 00007f398610c437 00007f398610c3c2 00007f398610c2f1 00007f398610c26c 00007f398610b522 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:58:57.605441 (+ 561us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":391787904}
I20251215 08:58:57.606518 22540 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.607432 22541 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.608143 22541 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.608824 21976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 238) took 1503 ms. Trace:
I20251215 08:58:57.606276 22543 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.609664 22543 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:58:57.610169 22543 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.610942 22494 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.621475 22494 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.622105 22494 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.622741 22543 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.623481 22543 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.624001 22543 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.611231 22541 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.617738 22544 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.626113 22543 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.626212 22544 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.626694 22543 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.627354 22543 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.629035 22494 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.629988 22543 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.634203 22228 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:58:57.634694 22228 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.630388 22544 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.637657 22228 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.609252 21976 rpcz_store.cc:276] 1215 08:58:56.105783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:56.107156 (+ 1373us) service_pool.cc:225] Handling call
1215 08:58:56.107460 (+ 304us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:56.107660 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:58:56.136908 (+ 29248us) log.cc:844] Serialized 14 byte log entry
1215 08:58:56.137958 (+ 1050us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:58:56.146274 (+ 8316us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:58:56.146345 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:57.608665 (+1462320us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:58:57.608763 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":116}
I20251215 08:58:57.638984 22544 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.641395 22543 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:58:57.642082 22543 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.644479 22543 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.645624 22228 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.647708 22228 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.646992 22517 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.646247 22549 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.647154 22519 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.660827 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.663897 22228 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.661669 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:57.664486 21413 raft_consensus.cc:3055] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:57.664948 21413 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:57.653643 22426 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.649313 22494 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.649573 22543 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.667651 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:57.668128 22543 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:57.670055 22543 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.670404 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.670583 22475 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.670974 21339 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:57.658037 22535 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.653764 22534 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.661618 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.662597 22518 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.666483 22519 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.664342 22228 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.673909 22228 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.651537 22517 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.662878 22536 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:57.646458 22544 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.672143 21413 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.672057 21332 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:57.671097 22475 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.671033 22549 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.675721 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.705705 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.704269 22228 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.708623 21339 raft_consensus.cc:2393] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:57.681988 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:57.709792 21415 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:57.712553 21413 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:57.713745 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:57.706485 22547 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.715734 22546 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.714389 22543 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.715883 22550 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.677920 22535 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.719185 22519 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:57.720470 21229 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.752s user 0.005s sys 0.011s
I20251215 08:58:57.673848 22494 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.677488 22534 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:57.720995 21229 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.753s user 0.005s sys 0.011s
I20251215 08:58:57.727564 22552 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.716590 22543 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.674010 22426 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.729645 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.730927 21336 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:57.716352 22546 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.725737 22535 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.675619 22518 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.736833 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:57.738453 22544 leader_election.cc:290] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.732103 22548 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.740609 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.741295 21406 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:57.742029 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:57.742672 22517 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:57.763995 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:57.764022 21410 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:57.764905 22517 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:58:57.765462 22517 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.743577 22549 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.754639 21407 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:57.753334 22538 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.767310 22538 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.720541 22547 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.762840 22476 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.731148 22550 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.777166 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.778873 22537 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.773845 22519 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:58:57.779526 22537 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.779613 22519 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.728238 22552 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.796376 22433 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.778311 22476 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.797221 22545 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.804661 22475 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:57.805186 22520 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.773458 22535 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.850893 21336 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:57.821154 22543 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.819465 22516 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.820838 22555 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.823493 22545 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.864280 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.714764 22536 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.868422 22398 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.820546 22534 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.761914 22548 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.819260 22475 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.835750 22544 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:57.835666 22426 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.809552 22433 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.874471 22398 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.873766 21413 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:57.877554 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:57.879897 22393 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:58:57.895823 22393 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.877617 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:57.901456 22519 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.901515 22546 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.901927 21408 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:57.822580 22535 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.826431 22520 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.854997 22513 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.895731 22556 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.900326 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:57.896037 22392 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.900827 22393 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.872084 22543 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:58:57.943156 22398 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.871009 22553 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:58:57.939375 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:57.917084 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:57.904728 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:57.941555 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:57.944586 21266 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:57.951206 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:57.951907 21979 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:57.894977 22517 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.025306 22393 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.901575 22476 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:57.905151 22473 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:57.945354 22392 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.915959 21978 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:57.896337 22475 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:57.873638 22516 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:57.894811 22544 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:58.067030 21412 consensus_meta.cc:287] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 1.932s user 0.020s sys 0.001s
I20251215 08:58:58.174787 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.270048 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.270170 21412 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.270784 21261 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.082804 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:58.272194 21412 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:58.138365 22393 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.168677 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.274644 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.150241 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:57.945225 21976 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.275238 21413 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:58.270682 21979 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.278098 22393 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.276854 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:57.949659 22546 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.280678 22398 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.282997 22398 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:58:58.044911 22496 consensus_meta.cc:287] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 1.925s user 0.023s sys 0.012s
I20251215 08:58:57.945582 22549 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.281503 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:32818 (request call id 1304) took 2139 ms. Trace:
I20251215 08:58:58.281822 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.283402 22398 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:57.945932 22518 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:58.291265 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.287271 21412 rpcz_store.cc:276] 1215 08:58:56.133495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:56.133605 (+ 110us) service_pool.cc:225] Handling call
1215 08:58:58.272582 (+2138977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:58:58.294275 21230 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:58.294382 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.295730 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.143604 22538 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.141152 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.046198 22550 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.146688 22515 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:58.066361 22537 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.297943 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1148) took 2195 ms. Trace:
I20251215 08:58:58.141384 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:58.046422 22494 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.166172 22536 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.138916 22476 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:58.148087 22535 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:57.889649 22555 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.316030 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:58.301316 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 422864us
I20251215 08:58:57.949899 22543 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.304960 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.316948 21413 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.308023 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.306476 21409 rpcz_store.cc:276] 1215 08:58:56.097722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:56.137122 (+ 39400us) service_pool.cc:225] Handling call
1215 08:58:56.137418 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:56.137606 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:58:57.866695 (+1729089us) log.cc:844] Serialized 14 byte log entry
1215 08:58:58.292194 (+425499us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:58:58.292258 (+ 64us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:58:58.292313 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:58.292401 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:58:58.292548 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1508,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":4460,"thread_start_us":425133,"threads_started":1}
I20251215 08:58:58.318077 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.292797 22546 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.309500 22550 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:58.310443 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.298511 22559 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:58.296945 22547 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.315541 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.174274 22552 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.318667 21413 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.300907 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.139509 22513 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.285494 22473 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.324900 21415 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.307991 22539 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.327503 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.330622 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.334309 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.297644 22549 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.362331 21412 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:58.362815 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.316784 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:57.955463 22553 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.324154 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.363780 21408 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.325935 22433 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.373468 22539 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:57.955732 22556 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.320456 22476 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:58.423389 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:58.441141 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:58.334359 22550 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.448870 21409 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.448998 21944 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:58.449138 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.371584 22559 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.456594 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.457213 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.364796 22545 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.403349 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.460042 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:58.447031 22539 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:58.450628 22433 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:58.316903 22515 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.451720 22552 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.374715 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.461618 21258 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.461922 21341 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:58.449965 22487 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:58.410161 22392 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.463531 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.459878 22398 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.450523 21406 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.463961 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.491357 22476 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:58.466470 21976 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.464457 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.494406 21261 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.494870 22398 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.371258 22548 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.364408 22520 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.465852 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:58.450420 22540 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:58.452641 22535 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:58.502836 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.498819 21307 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:58.499150 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.500489 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:58.374495 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:58:58.505873 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 58091us
I20251215 08:58:58.466589 21262 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.499241 21257 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.499724 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.466517 22552 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.502393 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.507900 22552 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:58.503902 21258 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.502264 21265 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:58:58.500619 22392 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.506655 21307 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:58.466781 22433 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.463824 22475 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:58.506497 21378 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:58.493372 22549 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:58:58.501971 21195 debug-util.cc:398] Leaking SignalData structure 0x7b0800041c80 after lost signal to thread 21198
I20251215 08:58:58.493849 22544 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:58.493377 22553 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.510145 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.522657 21266 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.509986 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.511979 22543 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.512226 22555 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:58.514839 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:58:58.500713 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.526034 22548 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.526321 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.515630 22563 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.526511 22548 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.526947 22563 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.527324 22548 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:58.527441 22563 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:58.500353 22540 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.512933 22392 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.514362 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.508850 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.493418 22487 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.509501 22496 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.526490 22559 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.507550 22535 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.507285 22516 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.500531 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:58.516896 22426 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.761698 21381 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:58:58.788887 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 277693us
I20251215 08:58:58.790942 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.694085 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.693653 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.794027 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:58.794401 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:58.522722 22499 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:58.788404 22496 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.796370 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:58.570648 22564 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20251215 08:58:58.798084 21381 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:58.798821 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:58.801062 21261 raft_consensus.cc:3055] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:58.801280 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:58.801937 21339 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:58.802142 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:58.525583 22546 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.583797 22473 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.801357 21261 raft_consensus.cc:740] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:58.692507 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.543144 22540 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.765344 22569 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:58.788946 22550 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.530335 22392 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:58.839121 21261 consensus_queue.cc:260] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.768985 22541 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:58:58.860462 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:58.710060 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.566594 22556 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.542704 22513 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.802698 22489 raft_consensus.cc:993] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:58.793823 22426 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:58.629164 22570 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.795956 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:58.796921 22534 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.792943 21980 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:58.961510 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.961562 21263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1129) took 2846 ms. Trace:
I20251215 08:58:58.820865 22499 raft_consensus.cc:697] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:58.942584 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.962234 21263 rpcz_store.cc:276] 1215 08:58:56.114622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:58.960565 (+2845943us) service_pool.cc:225] Handling call
1215 08:58:58.960914 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:58.961099 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:58:58.961127 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:58:58.961174 (+ 47us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:58:58.961211 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:58.961276 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:58:58.961381 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:58:58.797103 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:58.814456 22496 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:58.717067 22539 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.946244 22541 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.943357 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:58.948663 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 150360us
I20251215 08:58:58.952656 22392 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.686062 22548 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:58:58.956024 22398 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:58.986848 22398 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.803161 21974 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.943156 22569 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:58.987496 22392 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:58.943451 21257 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:58.625415 22515 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:58:58.960462 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 164573us
I20251215 08:58:59.006526 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:58.963757 22534 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.007977 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:58.993865 22513 raft_consensus.cc:1081] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:58:58.956344 22433 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.960060 22426 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.005455 22398 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:58.964818 22475 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:58:59.009768 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 211759us
I20251215 08:58:58.991703 21264 raft_consensus.cc:1240] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:58.985138 22570 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:58.953562 22552 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:59.021952 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 221569us
I20251215 08:58:59.014040 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:58.963860 22563 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:58.997157 22518 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:58.962934 22499 consensus_queue.cc:237] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.014483 22541 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.010905 22534 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.016705 22553 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.034024 21381 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:59.035039 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:58.995662 21262 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:59.015226 22550 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.028303 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.031942 22553 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.036310 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.027369 22549 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.037384 22574 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.048789 22574 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.037525 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1150) took 1373 ms. Trace:
I20251215 08:58:59.049470 22574 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.049937 21407 rpcz_store.cc:276] 1215 08:58:57.663734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:57.663996 (+ 262us) service_pool.cc:225] Handling call
1215 08:58:59.037348 (+1373352us) spinlock_profiling.cc:243] Waited 86.4 ms on lock 0x7b54001dfbd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:58:59.037450 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198681472}
I20251215 08:58:59.038055 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.038687 22573 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.053231 22573 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.025123 22570 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.041522 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.043167 21381 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.057790 21381 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:59.037074 22569 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.024264 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.043574 22535 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.046072 22539 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.046468 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.072510 21265 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.047381 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.059284 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.060137 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.060958 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1072) took 1119 ms. Trace:
I20251215 08:58:59.060885 21946 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.063724 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.044552 21978 raft_consensus.cc:1240] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
W20251215 08:58:59.078251 21379 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.549s (SENT) [suppressed 5 similar messages]
I20251215 08:58:59.081099 21406 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:59.063665 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:58:59.066780 22228 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.045220 22553 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.068075 22576 raft_consensus.cc:2804] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.070842 22548 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.027128 22518 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.045288 22475 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.075618 21407 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:59.082705 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.076471 21260 rpcz_store.cc:276] 1215 08:58:57.941469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:58.063622 (+122153us) service_pool.cc:225] Handling call
1215 08:58:58.064017 (+ 395us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:58.298515 (+234498us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:58:58.298551 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:58:58.298592 (+ 41us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:58:58.298632 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:59.060640 (+762008us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54002e59d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:58:59.060682 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:58:59.060803 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":525975424}
I20251215 08:58:59.079533 22549 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.079594 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.080372 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.089362 22228 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.065507 22487 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.056030 22573 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.090312 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.064766 22544 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.097646 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.074381 22563 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.104218 21979 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:59.102927 22228 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.105049 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.081110 22552 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.106212 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.094099 22576 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.111630 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.107261 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.111917 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:58:59.107309 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 49124.2us
I20251215 08:58:59.106972 21229 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:59.112385 21407 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:58:59.109582 21411 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:59.099315 21163 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45538 (request call id 148) took 3250 ms. Trace:
I20251215 08:58:59.109782 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:58:59.098457 22518 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.099279 22549 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.074062 22550 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20251215 08:58:59.098280 22475 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.099035 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.117475 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.118433 21258 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:59.118505 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.122677 21381 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.124902 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.105768 22544 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.105618 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.110931 21248 tablet_service.cc:1552] Processing DeleteTablet for tablet b3680af750ef47df88bf5be33eb18fc7 with delete_type TABLET_DATA_TOMBSTONED (TS beefe15185ac42ef9a20251007f39a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50692
I20251215 08:58:59.113206 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.116158 22511 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.114419 22576 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.117650 22563 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.137709 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.135983 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.130117 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.120859 22549 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.121074 22475 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.121196 22552 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.141243 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.158587 21977 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.152776 22511 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.160205 22511 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.136039 21972 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:59.157377 21407 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:59.125406 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.154676 21946 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.209873 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.210479 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.135550 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.212453 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.127763 22544 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.177115 21977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 257) took 1261 ms. Trace:
I20251215 08:58:59.152298 22563 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.223143 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.221728 21977 rpcz_store.cc:276] 1215 08:58:57.897406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:57.897548 (+ 142us) service_pool.cc:225] Handling call
1215 08:58:59.141086 (+1243538us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:58:59.159178 (+ 18092us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12268928}
I20251215 08:58:59.114123 21163 rpcz_store.cc:276] 1215 08:58:55.846095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:55.846360 (+ 265us) service_pool.cc:225] Handling call
1215 08:58:59.096371 (+3250011us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:58:57.627467 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:58:57.627992 (+ 525us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:58:57.628025 (+ 33us) write_op.cc:435] Acquired schema lock
1215 08:58:57.628040 (+ 15us) tablet.cc:660] Decoding operations
1215 08:58:57.628361 (+ 321us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:58:57.628411 (+ 50us) write_op.cc:641] Partition lock acquired for write op
1215 08:58:57.628457 (+ 46us) tablet.cc:683] Acquiring locks for 1 operations
1215 08:58:57.628641 (+ 184us) tablet.cc:699] Row locks acquired
1215 08:58:57.628660 (+ 19us) write_op.cc:260] PREPARE: finished
1215 08:58:57.628796 (+ 136us) write_op.cc:270] Start()
1215 08:58:57.628904 (+ 108us) write_op.cc:276] Timestamp: P: 1765789137628752 usec, L: 0
1215 08:58:57.628925 (+ 21us) op_driver.cc:348] REPLICATION: starting
1215 08:58:57.629595 (+ 670us) log.cc:844] Serialized 536 byte log entry
1215 08:58:57.893481 (+263886us) op_driver.cc:464] REPLICATION: finished
1215 08:58:58.529833 (+636352us) write_op.cc:301] APPLY: starting
1215 08:58:58.551309 (+ 21476us) tablet.cc:1365] starting BulkCheckPresence
1215 08:58:58.583001 (+ 31692us) tablet.cc:1368] finished BulkCheckPresence
1215 08:58:58.681442 (+ 98441us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:58:58.681986 (+ 544us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:58:58.682183 (+ 197us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1215 08:58:58.682243 (+ 60us) write_op.cc:312] APPLY: finished
1215 08:58:58.682476 (+ 233us) log.cc:844] Serialized 23 byte log entry
1215 08:58:58.682649 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:58:58.682722 (+ 73us) write_op.cc:454] Released schema lock
1215 08:58:58.682755 (+ 33us) write_op.cc:341] FINISH: Updating metrics
1215 08:58:58.683081 (+ 326us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:58:58.683111 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1506298,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":611148,"apply.run_cpu_time_us":2127,"apply.run_wall_time_us":159746,"num_ops":1,"prepare.queue_time_us":96503,"prepare.run_cpu_time_us":3340,"prepare.run_wall_time_us":7297,"raft.queue_time_us":116872,"raft.run_cpu_time_us":14790,"raft.run_wall_time_us":459216,"replication_time_us":242265,"thread_start_us":392863,"threads_started":3,"wal-append.queue_time_us":5084}]]}
I20251215 08:58:59.161558 22552 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.161908 22487 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:58:59.161346 21974 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46154
I20251215 08:58:59.209312 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.148453 22540 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
I20251215 08:58:59.234468 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.141479 22578 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.229377 21378 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.221210 22548 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.220708 22499 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:58:59.242447 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 87859.1us
I20251215 08:58:59.226744 22563 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.231865 22552 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.247366 21946 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.250871 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.250815 21335 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
W20251215 08:58:59.254299 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 131705us
I20251215 08:58:59.243291 21974 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:58:59.249648 22487 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:58:59.254395 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.242839 22583 raft_consensus.cc:2804] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.258843 21974 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.257733 21264 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:58:59.249465 22548 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.268101 22581 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.260803 22487 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.253161 22499 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.270829 21229 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.265411 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.271075 22583 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.278877 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.274595 22513 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.293301 22513 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.274600 22581 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.293680 22513 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.293938 22581 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.292191 22511 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.298293 21257 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:59.290585 21265 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:59.292745 22583 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.300809 22581 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.303018 22581 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.306397 22583 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.308621 22583 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.309631 22513 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.263931 22575 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.272639 22487 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:59.322611 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.323541 21339 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:59.327265 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.328552 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.330155 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.330264 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.330849 21335 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:59.331408 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.331634 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.332569 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.344578 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.312384 22575 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.348618 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.349382 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.340516 22584 tablet_replica.cc:333] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: stopping tablet replica
I20251215 08:58:59.350831 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.344388 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.361301 22228 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.361701 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:59.363155 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 106416us
I20251215 08:58:59.343060 22513 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.348002 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.365759 21381 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:59.365952 22228 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.366561 22514 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.353688 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.370023 22567 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:59.370123 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.370298 22519 consensus_queue.cc:1059] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:59.372413 21337 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.372711 22582 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.372869 22514 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.375337 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.375943 21337 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.376854 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.373242 22582 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.381179 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.389351 22539 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:59.381357 22499 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.381268 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:59.379501 22569 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.401974 21335 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.404151 21976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 263) took 1107 ms. Trace:
I20251215 08:58:59.409451 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.353752 22573 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.372546 22473 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:58:59.414031 22473 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.414358 22534 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.372519 22496 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.415334 22228 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.415869 22228 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.385253 22511 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:58:59.390683 21341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1351) took 1308 ms (client timeout 1520 ms). Trace:
I20251215 08:58:59.400584 22548 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.411839 22569 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.400007 22582 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.400465 21414 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:59.409495 21976 rpcz_store.cc:276] 1215 08:58:58.296994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:58.300976 (+ 3982us) service_pool.cc:225] Handling call
1215 08:58:59.403954 (+1102978us) spinlock_profiling.cc:243] Waited 89.8 ms on lock 0x7b5400628ed4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:58:59.404070 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206607616}
I20251215 08:58:59.424584 22575 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:58:59.391034 21195 debug-util.cc:398] Leaking SignalData structure 0x7b0800369da0 after lost signal to thread 21290
W20251215 08:58:59.425743 21341 rpcz_store.cc:269] 1215 08:58:58.082331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:58.082470 (+ 139us) service_pool.cc:225] Handling call
1215 08:58:59.390495 (+1308025us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400279cd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:58:59.390603 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436532480}
I20251215 08:58:59.378821 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.421329 22514 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.388939 22549 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.437135 21335 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.438675 22228 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.454064 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.442288 21378 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.442866 21408 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.443363 22475 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.444613 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:58:59.433333 21414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1155) took 1725 ms (client timeout 1907 ms). Trace:
I20251215 08:58:59.401111 22539 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.443732 22552 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.417763 22473 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.414937 22574 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.448084 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.465514 21336 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:58:59.449620 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.450286 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.466611 22584 raft_consensus.cc:2243] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:58:59.467425 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.454941 22228 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.468340 22228 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.419245 22576 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:59.447316 21195 debug-util.cc:398] Leaking SignalData structure 0x7b0800369dc0 after lost signal to thread 21291
I20251215 08:58:59.432917 22570 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.468771 22228 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.473296 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.469832 21976 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:58:59.411933 22563 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.471482 21977 raft_consensus.cc:1240] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.471530 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.454682 21259 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.479897 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.447647 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.464504 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.467137 22549 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.480090 21333 raft_consensus.cc:2393] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:59.461495 22541 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:59.487812 22541 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.474458 22496 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.481240 21381 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.493371 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.494058 22536 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:58:59.494439 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.495383 22539 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.495491 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.495795 22518 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.496605 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.494483 22536 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:59.497788 22586 raft_consensus.cc:3055] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.497859 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.498646 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.499189 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.499378 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.499792 21340 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:58:59.500020 21264 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.500420 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:58:59.461701 21414 rpcz_store.cc:269] 1215 08:58:57.707486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:57.708949 (+ 1463us) service_pool.cc:225] Handling call
1215 08:58:59.370946 (+1661997us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:58:59.433266 (+ 62320us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8619264}
I20251215 08:58:59.500694 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:59.502321 22573 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.498275 22586 raft_consensus.cc:740] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:59.502815 21408 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.503059 22475 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.503165 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:59.497804 22536 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:59.503789 21332 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.483507 21260 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.511019 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.512290 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.513653 21337 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:59.514647 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:59.515127 22228 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.393066 22567 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.484356 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.502858 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.484541 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.485121 22584 raft_consensus.cc:2272] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:58:59.533041 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.474346 21973 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.535377 22228 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.535763 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.536670 21413 raft_consensus.cc:2393] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:58:59.490777 22541 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.538892 21976 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:58:59.538849 22558 consensus_queue.cc:1059] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:58:59.539911 22558 raft_consensus.cc:3055] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.539865 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.540318 22558 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:59.491360 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 310) took 1028 ms. Trace:
I20251215 08:58:59.541244 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.541157 21978 rpcz_store.cc:276] 1215 08:58:58.462576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:58.527465 (+ 64889us) service_pool.cc:225] Handling call
1215 08:58:58.527756 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:59.080919 (+553163us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:59.491093 (+410174us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b54004709d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:58:59.491272 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1179966336}
I20251215 08:58:59.541801 21407 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:59.541996 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.542428 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.543557 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.498735 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.543906 22569 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.504258 21261 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.532636 22496 consensus_queue.cc:1059] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.012s
I20251215 08:58:59.491547 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.480511 21266 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.545220 22496 raft_consensus.cc:3055] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.545580 22496 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:59.472328 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.546334 22552 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.546778 21975 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:59.547205 22563 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.505167 21977 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:58:59.484146 21195 debug-util.cc:398] Leaking SignalData structure 0x7b080036a5e0 after lost signal to thread 21365
I20251215 08:58:59.521505 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.525384 21304 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.525664 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:59.526494 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.554158 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:59.531234 21946 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.467031 22544 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.537552 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.542587 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.556419 21407 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:58:59.544634 21978 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.504741 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.482460 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.549852 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.552057 21261 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.547817 22584 ts_tablet_manager.cc:1916] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:58:59.558693 21332 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
W20251215 08:58:59.566357 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.567332 21413 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:59.567849 21381 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:59.580497 22228 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:59.581105 22228 raft_consensus.cc:697] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:59.568832 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.582639 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.582832 21978 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.583654 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.584167 21261 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:58:59.585152 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.540606 22582 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.586858 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.589776 21161 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36408 (request call id 211) took 1939 ms. Trace:
I20251215 08:58:59.590457 22576 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.576228 21946 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.592072 21161 rpcz_store.cc:276] 1215 08:58:57.650579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:57.650775 (+ 196us) service_pool.cc:225] Handling call
1215 08:58:59.589718 (+1938943us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20251215 08:58:59.597491 21977 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:58:59.604450 21978 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.600814 22567 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:58:59.599977 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 62691.8us
I20251215 08:58:59.606506 22550 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.607240 21406 raft_consensus.cc:1240] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.607517 21304 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.548967 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.609064 21977 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
W20251215 08:58:59.612344 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 62977.8us
I20251215 08:58:59.564339 22569 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:58:59.553334 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.614357 21944 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:59.615468 21229 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:58:59.587662 22552 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:58:59.588999 22517 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:58:59.612780 22550 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.616161 22228 consensus_queue.cc:237] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.616884 22393 consensus_queue.cc:1059] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:59.623467 21944 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:59.621408 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.626224 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.622938 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.551725 21338 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:58:59.602327 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.629593 22393 raft_consensus.cc:3055] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.630326 22393 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:59.611925 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.515691 22586 consensus_queue.cc:260] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.632360 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.620954 22541 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
W20251215 08:58:59.635079 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 48565.5us
I20251215 08:58:59.629070 22583 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.616204 22548 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.612273 22534 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.638849 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:58:59.630359 22553 consensus_meta.cc:287] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.533s user 0.016s sys 0.002s
W20251215 08:58:59.627343 21195 debug-util.cc:398] Leaking SignalData structure 0x7b080026d6c0 after lost signal to thread 21439
I20251215 08:58:59.546424 22496 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.644359 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.624888 22517 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:58:59.655725 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.649852 22393 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.656781 22393 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.627101 22550 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.624308 22552 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:58:59.614204 22560 consensus_queue.cc:1059] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20251215 08:58:59.653954 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.591146 22558 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.655877 22517 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:59.646289 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.629300 22518 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.649386 22499 consensus_queue.cc:1059] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20251215 08:58:59.662842 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.664043 22570 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.664465 22574 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.651708 22576 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.666625 21260 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:58:59.653331 21265 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:59.642874 22473 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:59.667325 21407 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.649255 22583 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:59.647277 22548 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:59.656877 21414 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.673995 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.670879 22544 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.675064 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:58:59.704494 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 55016us
I20251215 08:58:59.654264 22553 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:58:59.670897 22426 consensus_meta.cc:287] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.650s user 0.020s sys 0.008s
I20251215 08:58:59.666227 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.669268 22552 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:58:59.708802 21413 raft_consensus.cc:1240] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
W20251215 08:58:59.701977 21195 debug-util.cc:398] Leaking SignalData structure 0x7b0800356200 after lost signal to thread 22004
I20251215 08:58:59.692160 22499 raft_consensus.cc:3055] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
W20251215 08:58:59.702476 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 69590us
I20251215 08:58:59.671886 22586 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.710297 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.711110 21411 raft_consensus.cc:2393] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:59.693138 21972 raft_consensus.cc:1240] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.705926 21381 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:59.694178 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.713409 22570 consensus_queue.cc:1059] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:58:59.714349 22570 raft_consensus.cc:3055] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.655754 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.713953 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.716053 21307 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.716313 22228 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:58:59.704227 22583 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.724234 21979 raft_consensus.cc:2393] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:58:59.703979 22548 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.707109 22558 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.731209 22228 raft_consensus.cc:697] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:58:59.729411 22582 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.729795 21414 raft_consensus.cc:1240] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.706656 22473 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:59.730649 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.714700 22570 raft_consensus.cc:740] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:59.703734 22576 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.715684 22499 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:59.736155 22393 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.739216 22487 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.726109 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.732005 21264 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.740302 22473 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.730686 22553 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.737320 21307 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.741297 21412 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:58:59.714978 22426 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.750272 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.750999 21258 raft_consensus.cc:2393] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:59.751407 22426 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.751760 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.752014 21409 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.752424 21258 raft_consensus.cc:2393] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:59.732728 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.735136 22228 consensus_queue.cc:237] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:58:59.746434 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.753916 21415 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: []
I20251215 08:58:59.752439 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.747876 21973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 303) took 1432 ms. Trace:
I20251215 08:58:59.754691 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.755275 22393 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.756121 21971 raft_consensus.cc:1240] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.756325 22499 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.757597 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.757754 22539 consensus_queue.cc:1059] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:58:59.758721 22510 raft_consensus.cc:3055] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.759213 22510 raft_consensus.cc:740] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:59.757064 21258 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:58:59.761888 21258 raft_consensus.cc:606] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Received request to transfer leadership to TS 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:58:59.758778 22539 raft_consensus.cc:3055] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.763052 22539 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:59.763628 22541 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.764310 22541 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.766290 22541 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.767401 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:58:59.733597 22593 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.755708 22550 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.766914 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.772107 22487 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.769456 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.773589 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.773953 22576 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.774240 21406 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:58:59.774430 22569 raft_consensus.cc:3055] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.774818 22569 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:59.772876 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.773610 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.757629 22499 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.775183 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.784127 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.784649 21979 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:58:59.777310 22569 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.778394 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:58:59.778405 22536 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.785419 22548 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.788432 21972 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.775583 22570 consensus_queue.cc:260] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.789376 22548 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.780272 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.781639 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.760074 22510 consensus_queue.cc:260] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.770025 22539 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.773622 21408 raft_consensus.cc:2393] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:59.785583 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.811925 21978 raft_consensus.cc:2393] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:58:59.808821 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.810575 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.810915 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.814261 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.813293 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:58:59.815864 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.816166 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.817689 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.819062 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:58:59.755378 21973 rpcz_store.cc:276] 1215 08:58:58.315218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:58.332417 (+ 17199us) service_pool.cc:225] Handling call
1215 08:58:59.403404 (+1070987us) spinlock_profiling.cc:243] Waited 2.38 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:58:59.403569 (+ 165us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:59.543633 (+140064us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:59.747628 (+203995us) spinlock_profiling.cc:243] Waited 65.4 ms on lock 0x7b5400570154. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:58:59.747820 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155862144}
I20251215 08:58:59.820072 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.821780 21415 raft_consensus.cc:2393] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:59.821848 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.812177 22548 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:58:59.749832 21974 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.835326 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 305) took 1465 ms. Trace:
I20251215 08:58:59.835964 22510 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.790100 22570 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.814945 22586 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.818202 22429 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251215 08:58:59.847311 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:58:59.851210 21978 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:58:59.790434 22550 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.837985 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.842854 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.841467 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:59.852273 21378 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.858178 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.862888 22393 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.846467 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.848134 22552 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.848430 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.864687 21415 raft_consensus.cc:1813] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Handling vote request from an unknown peer beefe15185ac42ef9a20251007f39a56
I20251215 08:58:59.865228 21415 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:59.849525 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.864873 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:59.849713 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.865394 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.870779 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.873327 22548 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.873590 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:59.873723 22475 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.874620 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.875030 22496 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.875725 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:59.876645 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
W20251215 08:58:59.877180 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.845099 22575 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.842214 22518 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.874799 21408 raft_consensus.cc:2393] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:58:59.880939 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:58:59.880812 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.882382 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.882923 21413 raft_consensus.cc:606] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:58:59.882288 21163 catalog_manager.cc:5654] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: config changed from index 2 to 3, term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to <none>, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:58:59.888211 21164 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36406 (request call id 225) took 3948 ms. Trace:
I20251215 08:58:59.888610 21164 rpcz_store.cc:276] 1215 08:58:55.940054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:55.940206 (+ 152us) service_pool.cc:225] Handling call
1215 08:58:59.888152 (+3947946us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137323,"reported_tablets":2}
I20251215 08:58:59.891224 21409 raft_consensus.cc:1240] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.890525 21259 consensus_queue.cc:237] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.893950 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:59.894551 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.893362 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.860044 22539 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.898814 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.899106 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:58:59.877931 21414 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.877199 21976 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.877750 22575 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:58:59.902143 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:59.844462 22517 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:58:59.903715 21337 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.905408 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:58:59.900689 21379 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:59.910243 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:58:59.911455 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.914248 21339 raft_consensus.cc:1240] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.893085 22586 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:58:59.864570 22514 consensus_queue.cc:1059] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20251215 08:58:59.926026 21336 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
W20251215 08:58:59.933274 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.914543 22575 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.936847 21415 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:58:59.906884 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:58:59.968482 21406 raft_consensus.cc:2393] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:58:59.918644 21410 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:58:59.952801 22514 raft_consensus.cc:3055] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:58:59.977059 21379 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:58:59.978189 21379 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:59.979630 21379 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:59.959125 22583 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:58:59.960708 22552 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:58:59.961403 22536 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:58:59.961711 21974 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:58:59.965365 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:58:59.973817 22393 consensus_queue.cc:1059] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20251215 08:59:00.001612 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.001716 22593 raft_consensus.cc:3055] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.002362 22593 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:58:59.940022 21336 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:58:59.846328 21971 rpcz_store.cc:276] 1215 08:58:58.369992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:59.400353 (+1030361us) service_pool.cc:225] Handling call
1215 08:58:59.753158 (+352805us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:58:59.756553 (+ 3395us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:58:59.835054 (+ 78501us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54003ed850. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:58:59.835266 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":810430336}
I20251215 08:58:59.982051 22558 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:58:59.984573 22576 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:58:59.985610 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:58:59.986552 22569 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:58:59.986898 22514 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:58:59.990450 21411 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:00.008638 22558 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:58:59.992812 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:58:59.923017 22535 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.208554 21265 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:00.005862 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:00.015311 21261 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:00.209872 21261 raft_consensus.cc:606] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:00.012650 22398 consensus_queue.cc:1059] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:00.210842 22398 raft_consensus.cc:3055] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.211136 22398 raft_consensus.cc:740] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:00.211645 22574 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:00.098699 22589 log.cc:927] Time spent T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20251215 08:59:00.037793 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:00.003387 22593 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.044288 22558 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.035742 21264 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:00.112116 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.099107 22514 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.207386 22473 consensus_queue.cc:1059] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:00.208585 21333 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:00.212797 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:00.199462 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.213913 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.211819 22398 consensus_queue.cc:260] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.215579 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:00.227566 21337 raft_consensus.cc:1240] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:00.236771 22228 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.233675 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:00.236680 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 127804us
I20251215 08:59:00.238667 22398 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.239199 22593 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.239604 21411 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:00.240258 21411 raft_consensus.cc:606] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:00.240628 21411 raft_consensus.cc:614] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251215 08:59:00.227725 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 234973us
I20251215 08:59:00.242931 22228 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:00.243796 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.250669 22517 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:00.251892 22517 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:00.260033 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:00.263301 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:00.212116 22574 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.284456 21335 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:00.274536 21262 raft_consensus.cc:3055] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.274636 21332 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:00.259049 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.289489 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.272606 21334 raft_consensus.cc:1240] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:00.286259 22514 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.291103 21307 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.252512 22575 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.295392 22393 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.296844 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.298460 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.292024 21260 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:00.301249 22569 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:00.301084 22593 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:00.308643 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 106752us
I20251215 08:59:00.310163 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:00.313519 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:00.313939 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:00.315032 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.304368 22583 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.296002 22496 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:00.318025 21972 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:59:00.319188 21260 raft_consensus.cc:606] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Received request to transfer leadership to TS 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:00.309468 22599 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.230s user 0.003s sys 0.005s
I20251215 08:59:00.328917 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:59:00.329850 22496 consensus_queue.cc:1059] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:00.330693 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:00.330806 22496 raft_consensus.cc:3055] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.287431 21262 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:00.326596 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:00.328114 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:00.334154 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.334463 22393 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:00.332037 22398 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:00.302999 22575 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:00.334757 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 91045us
I20251215 08:59:00.334904 22393 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:00.336818 22393 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.331146 22496 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:00.309597 22593 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:00.338451 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.333781 21261 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.309278 22569 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:00.325747 22499 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:00.331835 22570 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:00.324746 21263 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:00.341981 22429 consensus_queue.cc:1059] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251215 08:59:00.342451 22593 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.340900 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.342623 22569 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.343266 22499 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:00.344246 22570 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:00.365446 21264 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:00.366508 22499 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.368856 21307 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.372658 22574 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:00.348238 21262 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.372483 22548 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:00.379042 22548 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:00.379238 21262 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.379645 22548 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.372637 22570 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.367704 22496 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.394809 22575 raft_consensus.cc:3055] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.395385 22575 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:00.396667 22575 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.397964 22496 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.398173 22575 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.400488 22429 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:00.400875 22429 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:00.401307 22429 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.401736 22558 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.405027 22558 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:00.407298 21379 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.408445 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.410076 22574 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:00.410511 22574 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:00.410866 22574 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.414654 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.415472 22534 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:00.415556 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 201705us
I20251215 08:59:00.416703 21379 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.419083 22578 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:00.423826 21414 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:00.425568 22558 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:00.426009 22558 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:00.426630 22558 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.428166 22578 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.429160 22519 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.430941 21379 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:00.432821 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.434528 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:00.437819 21974 raft_consensus.cc:1240] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:00.441354 21262 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
W20251215 08:59:00.441936 22516 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20251215 08:59:00.416602 22393 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:00.445425 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.446332 22393 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.446381 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:00.447090 21263 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:00.447697 21264 raft_consensus.cc:1275] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:00.448135 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:00.449416 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:00.450138 22519 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:00.450507 22519 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:00.450796 22578 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:00.450832 22519 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.455825 22578 raft_consensus.cc:993] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:59:00.456341 22578 raft_consensus.cc:1081] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:59:00.458438 21974 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46158
I20251215 08:59:00.459020 21974 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:00.459437 21974 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.461263 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:00.462116 21946 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:00.462348 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:00.463621 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 94953.9us
I20251215 08:59:00.464143 21976 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:00.464342 21307 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.465476 22496 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.467023 22544 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.467643 22544 raft_consensus.cc:697] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:00.468336 21974 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.469200 22583 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.469732 22583 raft_consensus.cc:697] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:00.470458 21974 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.470646 22583 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.480041 21261 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:00.480640 22578 consensus_queue.cc:1059] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:00.482493 22578 raft_consensus.cc:3055] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.482848 22578 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.483599 22578 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.484898 22578 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:00.485948 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 55091.7us
I20251215 08:59:00.466001 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.488188 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:00.488703 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.488749 21261 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.495082 21379 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.496414 22393 consensus_queue.cc:1059] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:00.497661 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.508529 22393 raft_consensus.cc:3055] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.502630 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:00.508937 22393 raft_consensus.cc:740] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:00.509610 22393 consensus_queue.cc:260] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.512449 21379 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.513751 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.516297 21379 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.519830 21379 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.520897 22228 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:00.521281 22228 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:00.521584 22228 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:00.522715 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.523953 21261 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:00.523851 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.525450 21414 raft_consensus.cc:2393] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:00.525604 21411 raft_consensus.cc:1240] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251215 08:59:00.526432 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.526672 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:00.527086 21413 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:00.527629 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.528317 21410 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:00.528846 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.529155 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.529448 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.529860 21409 raft_consensus.cc:2393] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:00.530236 21407 raft_consensus.cc:3055] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.528878 22594 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.530676 21407 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.531030 22594 raft_consensus.cc:697] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:00.531949 22594 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.548728 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.529927 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.549960 21413 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:00.531770 21407 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.537632 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.551486 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.551797 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:00.552165 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.552606 21340 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:00.528338 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.540014 22473 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:00.553476 21415 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:00.540482 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:00.553625 22473 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.543448 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.555927 22473 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.556738 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.546187 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:00.557658 21407 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.546416 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.540594 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.563120 22473 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:00.563697 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.564275 21232 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:00.566100 22228 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.544389 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.563562 21335 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:00.551731 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.566532 22473 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.491288 22575 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.569593 22393 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.468552 22544 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.571612 21232 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:00.498858 22496 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.580539 21341 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:00.584699 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.515177 22536 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:00.507157 22514 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:00.517724 22563 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:00.518831 22607 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.570359 22426 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.518533 22582 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.494880 22516 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20251215 08:59:00.581454 22567 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.590834 21407 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:00.591475 22228 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:00.593703 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:00.600885 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:00.601266 21946 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.604373 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.605748 21407 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:00.607074 21415 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251215 08:59:00.608536 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:00.609197 21415 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:00.582291 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.611312 21232 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:00.611995 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.613076 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:00.609817 21336 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:00.616384 21379 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.618152 22398 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.589629 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.588856 22563 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:00.595254 22609 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.622802 22228 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.623168 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.622325 21232 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
W20251215 08:59:00.624814 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 127245us
I20251215 08:59:00.597368 22536 raft_consensus.cc:697] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.630028 21232 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:00.630314 22228 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:00.631788 21379 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.594540 22514 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.619923 22539 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.615613 22552 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.633342 22604 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:00.635751 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.637284 22584 ts_tablet_manager.cc:1929] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:59:00.635738 21411 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:00.626245 22563 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.638114 22429 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.642892 21334 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:00.643287 22519 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.633719 22496 consensus_queue.cc:1059] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251215 08:59:00.644963 22582 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.645334 22609 consensus_queue.cc:1059] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:00.649551 21379 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.653033 21232 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.625975 22475 consensus_queue.cc:1059] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20251215 08:59:00.628912 22574 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.655359 22393 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.634557 22605 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.662415 22584 log.cc:1199] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-0-root/wals/b3680af750ef47df88bf5be33eb18fc7
I20251215 08:59:00.668785 22398 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.668952 22228 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.670377 21379 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.670697 21232 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.655064 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.660317 22604 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
W20251215 08:59:00.674502 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 70633.3us
I20251215 08:59:00.705521 21379 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:00.677975 22612 log.cc:927] Time spent T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251215 08:59:00.678336 22552 raft_consensus.cc:3055] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:00.679217 22429 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:00.707161 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 52176.8us
I20251215 08:59:00.707638 22393 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.707379 21379 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:00.704824 21232 raft_consensus.cc:2686] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251215 08:59:00.712513 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:00.714003 22398 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:00.712739 22228 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.662997 22569 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.714942 22228 raft_consensus.cc:697] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.715307 21307 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.717509 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.703122 22575 raft_consensus.cc:3055] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:00.711486 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.709316 22604 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.717540 21406 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:00.741747 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.718755 21379 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.710475 22534 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.709537 22607 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.753561 22393 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:00.755486 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.777132 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.743659 22429 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.777259 22393 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.798331 22393 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:00.682930 22548 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.741353 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.710775 22613 raft_consensus.cc:3055] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:00.710675 22552 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:00.675985 22499 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:00.807142 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 88445.8us
I20251215 08:59:00.808305 22393 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.805596 22228 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.663992 22593 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.709069 22558 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.719534 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:00.720661 22575 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:00.745699 22473 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.745816 22496 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.745234 22544 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.800485 22609 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.703271 22536 consensus_queue.cc:237] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.807109 22613 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:00.848745 22613 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.849689 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:00.850317 21261 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.850728 22613 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.759250 22570 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.817636 22603 raft_consensus.cc:993] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56: : Instructing follower 195a56ac72a84018880bfb3bf6e6ef1d to start an election
I20251215 08:59:00.805207 22534 raft_consensus.cc:697] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.813330 22586 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.826948 22475 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.830392 22574 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:00.830890 22426 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.831923 22593 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:00.825488 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:00.833076 22611 log.cc:927] Time spent T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
W20251215 08:59:00.863420 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.863974 22569 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:00.836023 22584 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50692 (request call id 976) took 1739 ms. Trace:
I20251215 08:59:00.859247 22534 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.860514 22603 raft_consensus.cc:1081] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Signalling peer 195a56ac72a84018880bfb3bf6e6ef1d to start an election
I20251215 08:59:00.875311 21261 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:00.881354 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:00.868358 22496 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:00.888192 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 111184us
I20251215 08:59:00.889528 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.872624 22514 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.876729 22519 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:00.887164 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 77361.7us
W20251215 08:59:00.886067 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 86576.4us
I20251215 08:59:00.884955 22574 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:00.892964 21307 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.893167 21379 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:00.893157 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 174115us
W20251215 08:59:00.892112 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:00.899292 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 157023us
I20251215 08:59:00.909667 21232 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:00.910415 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:00.892478 22609 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:00.872638 22552 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:00.919026 21379 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.915336 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:00.924093 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.931318 22519 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.932142 22519 raft_consensus.cc:697] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.933480 22519 consensus_queue.cc:237] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.899883 22593 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.891656 22496 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.896268 22569 consensus_queue.cc:1059] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20251215 08:59:00.942734 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:00.943572 21338 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:00.928735 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:00.900285 22574 raft_consensus.cc:697] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.907626 22426 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.949517 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.950285 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.953186 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:00.962459 21976 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:00.960708 22582 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.935631 22473 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:00.872697 22584 rpcz_store.cc:276] 1215 08:58:59.097351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:59.099286 (+ 1935us) service_pool.cc:225] Handling call
1215 08:58:59.314788 (+215502us) ts_tablet_manager.cc:1151] Deleting tablet b3680af750ef47df88bf5be33eb18fc7
1215 08:58:59.316656 (+ 1868us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1215 08:59:00.208857 (+892201us) tablet_metadata.cc:641] Metadata flushed
1215 08:59:00.636961 (+428104us) tablet_metadata.cc:641] Metadata flushed
1215 08:59:00.835704 (+198743us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac90f2 00007f398d99a419 00007f398caf3108 00007f398cbbb99e 00007f398cbbe969 00007f398cbe772e 00007f398cbe76f9 00007f398cbe76b5 00007f398cbd8976
1215 08:59:00.835926 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":111712,"spinlock_wait_cycles":27317504,"tablet-delete.queue_time_us":158301,"thread_start_us":83463,"threads_started":1}
I20251215 08:59:00.949445 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.956899 22563 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.947021 22548 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.959563 22574 consensus_queue.cc:237] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.986449 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:00.887418 22575 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.967083 22398 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:00.991773 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:00.969497 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 49427.7us
I20251215 08:59:00.994930 21379 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:00.950223 22499 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.998827 22552 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.976579 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:01.000530 22606 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:00.985615 21406 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.001036 22606 raft_consensus.cc:697] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:00.983654 21410 raft_consensus.cc:1240] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:00.980979 22496 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.946306 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:00.980533 22570 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:00.966677 22593 raft_consensus.cc:697] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:00.979826 22539 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.006742 22593 consensus_queue.cc:237] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:00.955211 22558 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:00.966939 22569 raft_consensus.cc:3055] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:01.009600 22569 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:01.010627 22569 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:01.011996 22569 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:01.013828 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:01.016911 21212 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 7edcdb5749ae4e59ab17648fe97390e2
W20251215 08:59:01.017401 21212 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 7edcdb5749ae4e59ab17648fe97390e2
I20251215 08:59:01.018814 21163 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45538 (request call id 149) took 1530 ms. Trace:
I20251215 08:59:00.979462 21151 catalog_manager.cc:4985] TS beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): tablet b3680af750ef47df88bf5be33eb18fc7 (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:59:01.022940 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.026527 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:01.027521 21409 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.029683 21161 catalog_manager.cc:5654] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:01.001968 22606 consensus_queue.cc:237] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:01.038897 21161 catalog_manager.cc:5654] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:01.041144 21161 catalog_manager.cc:5654] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:01.036341 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.036692 21340 raft_consensus.cc:1240] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.019124 21163 rpcz_store.cc:276] 1215 08:58:59.488861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:59.488972 (+ 111us) service_pool.cc:225] Handling call
1215 08:59:01.018755 (+1529783us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:00.258826 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:00.320524 (+ 61698us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:00.320557 (+ 33us) write_op.cc:435] Acquired schema lock
1215 08:59:00.320576 (+ 19us) tablet.cc:660] Decoding operations
1215 08:59:00.320891 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:00.320945 (+ 54us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:00.320977 (+ 32us) tablet.cc:683] Acquiring locks for 1 operations
1215 08:59:00.321189 (+ 212us) tablet.cc:699] Row locks acquired
1215 08:59:00.321214 (+ 25us) write_op.cc:260] PREPARE: finished
1215 08:59:00.321354 (+ 140us) write_op.cc:270] Start()
1215 08:59:00.321467 (+ 113us) write_op.cc:276] Timestamp: P: 1765789140321306 usec, L: 0
1215 08:59:00.321491 (+ 24us) op_driver.cc:348] REPLICATION: starting
1215 08:59:00.322117 (+ 626us) log.cc:844] Serialized 565 byte log entry
1215 08:59:00.626822 (+304705us) op_driver.cc:464] REPLICATION: finished
1215 08:59:00.731477 (+104655us) write_op.cc:301] APPLY: starting
1215 08:59:00.756632 (+ 25155us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:00.807069 (+ 50437us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:00.823970 (+ 16901us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:00.865586 (+ 41616us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:00.872160 (+ 6574us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1215 08:59:00.872221 (+ 61us) write_op.cc:312] APPLY: finished
1215 08:59:00.887463 (+ 15242us) log.cc:844] Serialized 23 byte log entry
1215 08:59:00.894168 (+ 6705us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:00.897305 (+ 3137us) write_op.cc:454] Released schema lock
1215 08:59:00.898312 (+ 1007us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:00.939915 (+ 41603us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:00.942304 (+ 2389us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":144658,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":80422,"apply.run_cpu_time_us":2684,"apply.run_wall_time_us":213664,"num_ops":1,"prepare.queue_time_us":371858,"prepare.run_cpu_time_us":12005,"prepare.run_wall_time_us":169840,"raft.queue_time_us":146946,"raft.run_cpu_time_us":5903,"raft.run_wall_time_us":78513,"replication_time_us":280444,"thread_start_us":506185,"threads_started":4,"wal-append.queue_time_us":116554}]]}
I20251215 08:59:01.050683 22620 consensus_queue.cc:1059] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:01.053593 22620 raft_consensus.cc:3055] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:01.053970 22620 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:01.054813 22620 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.056008 22620 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:00.974814 22604 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:01.071789 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 162188us
I20251215 08:59:01.086447 22398 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50692 (request call id 978) took 1191 ms. Trace:
I20251215 08:59:01.086812 21232 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:01.086395 22563 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:01.075518 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:01.138144 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.086956 22398 rpcz_store.cc:276] 1215 08:58:59.887868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:58:59.888889 (+ 1021us) service_pool.cc:225] Handling call
1215 08:58:59.896843 (+ 7954us) log.cc:844] Serialized 501 byte log entry
1215 08:59:01.079338 (+1182495us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86148,"raft.queue_time_us":88729,"raft.run_cpu_time_us":948,"raft.run_wall_time_us":949,"thread_start_us":304238,"threads_started":2,"wal-append.queue_time_us":352395}
I20251215 08:59:01.140300 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4f1c639a483f4827997eae6f3a6e375d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:59:01.077956 22567 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.143401 21338 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:01.088645 22575 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:01.189873 22621 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.216s user 0.000s sys 0.000s
I20251215 08:59:01.174734 22473 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:01.206319 22604 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:01.215559 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:01.220247 22578 consensus_meta.cc:287] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.730s user 0.008s sys 0.004s
I20251215 08:59:01.229317 22609 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:01.241148 22605 consensus_meta.cc:287] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.538s user 0.005s sys 0.007s
I20251215 08:59:01.241575 22605 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:01.249035 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:01.249617 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 109717us
W20251215 08:59:01.250308 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 321476us
I20251215 08:59:01.255785 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.257789 21336 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:01.258209 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.259135 21337 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.267645 22618 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:01.269320 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.269841 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.270629 21334 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.272663 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:01.272994 22575 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:01.285580 22575 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.286808 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:01.287453 22575 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:01.287928 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:01.288257 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:01.289381 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:01.290000 21264 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:01.291317 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:01.273692 21263 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:01.276577 21971 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:01.277416 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:01.265843 22473 raft_consensus.cc:697] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:01.278782 21260 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:01.279841 21259 raft_consensus.cc:1240] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.280540 21266 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.260612 22625 consensus_queue.cc:1059] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251215 08:59:01.296254 21412 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:01.301019 21340 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:01.307365 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:01.315579 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.317095 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.321651 21337 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:01.321942 21258 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:01.322194 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.311703 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:01.323449 21334 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:01.323328 22473 consensus_queue.cc:237] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:01.323478 21972 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:01.327222 22625 raft_consensus.cc:3055] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
W20251215 08:59:01.322566 21944 leader_election.cc:336] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.711s (SENT)
I20251215 08:59:01.327580 22625 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:01.327819 21944 leader_election.cc:304] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:01.328403 22625 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:01.328820 21304 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:01.328900 22550 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:01.330098 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.331801 21265 raft_consensus.cc:2393] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
W20251215 08:59:01.335529 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.345710 21340 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:01.342446 22575 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:01.389609 22575 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:01.389988 22575 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:01.346822 22514 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:01.389629 22625 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:01.401696 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.403180 21340 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:01.404481 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.405277 21334 raft_consensus.cc:2393] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:01.413491 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.419579 21258 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:01.424153 21337 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:01.354835 22582 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20251215 08:59:01.399037 22578 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:01.445874 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.441751 22607 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:01.455755 22607 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:01.458329 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.401696 22514 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.460024 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.461179 21337 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:01.462036 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:01.463393 21972 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:01.461680 22607 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:01.447677 22578 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.466953 22578 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:01.468354 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:01.468976 22514 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:01.471128 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:01.471896 21265 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:59:01.475179 21381 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:01.477697 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.484628 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.485261 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.485910 21335 raft_consensus.cc:2376] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2. Re-sending same reply.
I20251215 08:59:01.486002 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.486361 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.486771 21341 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.487159 21337 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
W20251215 08:59:01.489605 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.491888 21336 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
W20251215 08:59:01.497119 21230 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.020s sys 0.014s
W20251215 08:59:01.497613 21230 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.020s sys 0.014s
W20251215 08:59:01.498056 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 561910us
I20251215 08:59:01.509646 21332 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:01.507668 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:01.534878 21332 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251215 08:59:01.535187 21332 pending_rounds.cc:85] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 1
I20251215 08:59:01.535400 21332 pending_rounds.cc:107] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251215 08:59:01.535635 21332 raft_consensus.cc:2889] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:01.512902 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:01.513392 22228 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.037s sys 0.083s
W20251215 08:59:01.540169 22228 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s user 0.037s sys 0.083s
I20251215 08:59:01.514034 21230 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:01.520066 21341 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:01.518947 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:01.544512 21265 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:01.547959 21415 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.545708 21409 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:01.547159 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:01.525761 21412 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:01.524390 21335 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.530030 21336 raft_consensus.cc:1275] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:01.563004 21340 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
W20251215 08:59:01.557399 21261 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.008s sys 0.028s
I20251215 08:59:01.564075 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:01.564493 21261 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.620s user 0.009s sys 0.028s
I20251215 08:59:01.565017 22228 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:01.508874 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.522571 21946 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:01.567075 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 44584.6us
I20251215 08:59:01.560492 21407 raft_consensus.cc:1240] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.567996 21978 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:01.531010 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.569813 21339 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:01.561262 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:01.571239 21412 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:01.558894 21230 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:01.573269 22398 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.573899 21379 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:01.526751 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.567101 21333 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.557705 21415 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:32818
W20251215 08:59:01.562404 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 54837us
I20251215 08:59:01.579175 21415 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:01.579550 21415 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:01.543022 22605 raft_consensus.cc:2749] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.544668 22558 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:01.551458 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.540737 21972 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:01.584458 21406 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.518746 22630 consensus_queue.cc:1059] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20251215 08:59:01.558579 22578 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:01.582417 22620 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:01.582281 22582 consensus_queue.cc:1059] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251215 08:59:01.581210 22593 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:01.593163 22626 raft_consensus.cc:2804] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:01.514019 22604 consensus_queue.cc:1059] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20251215 08:59:01.584800 22426 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20251215 08:59:01.598063 21412 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:01.599838 21307 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:01.553360 22539 consensus_queue.cc:1059] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20251215 08:59:01.610666 21229 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:01.583889 21230 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:01.620862 21230 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:01.622164 21230 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:01.599530 22558 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.618458 21414 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:01.604382 22563 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:01.604578 22607 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:01.605221 22593 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:01.637655 21338 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.647430 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.649212 22635 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.640169 22514 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:01.653208 22398 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.613044 22620 raft_consensus.cc:697] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:01.619150 22626 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:01.601771 22569 consensus_queue.cc:1059] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20251215 08:59:01.606695 22609 consensus_queue.cc:1059] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
W20251215 08:59:01.663098 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 59078.2us
I20251215 08:59:01.662665 21415 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:01.718922 21415 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:01.665655 22514 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:01.661619 22567 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.660920 22593 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:01.720449 21161 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.026s sys 0.051s
I20251215 08:59:01.676713 22626 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:01.722870 22228 raft_consensus.cc:3055] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:01.729059 22228 raft_consensus.cc:740] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:01.727039 21161 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.669s user 0.026s sys 0.051s
I20251215 08:59:01.661928 22607 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.716464 22535 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:01.753913 21944 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.528s (SENT)
I20251215 08:59:01.759254 21944 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:01.766475 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.763187 22228 consensus_queue.cc:260] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:01.726265 22514 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:01.789052 22636 log.cc:927] Time spent T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20251215 08:59:01.729120 22563 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
W20251215 08:59:01.806526 22643 log.cc:927] Time spent T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20251215 08:59:01.809628 22638 log.cc:927] Time spent T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20251215 08:59:01.792416 22620 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.800103 22583 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.766330 22535 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:01.824739 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:01.786520 22609 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:01.850189 21409 raft_consensus.cc:2955] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:01.841876 22518 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.009s sys 0.021s
W20251215 08:59:01.888576 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 265308us
I20251215 08:59:01.890895 21230 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:01.866622 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:01.866802 22535 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:01.867913 22633 raft_consensus.cc:3055] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
W20251215 08:59:01.898777 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 172559us
I20251215 08:59:01.865669 22569 raft_consensus.cc:3055] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Stepping down as leader of term 2
W20251215 08:59:01.887120 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 287331us
W20251215 08:59:01.867870 22647 log.cc:927] Time spent T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20251215 08:59:01.902818 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.861403 22625 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251215 08:59:01.895200 22518 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.009s sys 0.022s
I20251215 08:59:01.894258 22574 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:01.855403 22609 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.906270 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.906622 21230 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:01.909955 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:01.907166 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.911441 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.921010 21333 raft_consensus.cc:2393] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:01.911538 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:01.912267 21337 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:01.897046 22624 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:01.901510 22569 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:01.887977 22575 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.922683 21335 raft_consensus.cc:1813] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Handling vote request from an unknown peer beefe15185ac42ef9a20251007f39a56
I20251215 08:59:01.908308 21411 raft_consensus.cc:1240] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.901229 22633 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:01.917124 22607 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:01.928238 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:01.904790 22516 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:01.929916 22473 raft_consensus.cc:2804] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251215 08:59:01.969614 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:01.929735 21263 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:01.931572 21335 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.939870 21265 raft_consensus.cc:1275] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:01.912330 22558 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:01.946054 22398 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.022782 22398 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.024565 22398 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:01.945802 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:01.943773 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 189901us
I20251215 08:59:01.943466 22605 raft_consensus.cc:3055] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
W20251215 08:59:01.959357 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 45507.1us
I20251215 08:59:01.945603 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:02.027019 21413 raft_consensus.cc:2393] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:01.953621 21412 raft_consensus.cc:1240] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.959901 22630 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:01.963321 21407 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:01.931525 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:01.958559 21340 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:01.917614 22574 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:01.907018 22563 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251215 08:59:01.998350 21307 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:01.968865 21261 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:02.020071 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:01.962957 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.018573 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:01.932371 22624 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:02.029675 22652 log.cc:927] Time spent T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20251215 08:59:02.030278 22664 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:02.030982 22605 raft_consensus.cc:740] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:01.996428 22473 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:02.031141 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:02.053802 21972 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:02.031524 22609 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.031980 21334 raft_consensus.cc:2368] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:02.056043 22633 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:02.057346 22633 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.046487 22536 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20251215 08:59:02.058454 22603 raft_consensus.cc:3055] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.012982 21230 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:02.058823 22603 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:01.996793 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.059626 22603 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.061751 22539 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.061509 22605 consensus_queue.cc:260] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.038453 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:02.062758 22605 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:02.047236 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 49010.7us
I20251215 08:59:02.063977 22569 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.064534 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.064761 22626 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.065470 22609 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.065582 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:02.066654 21307 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:02.067270 22626 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.067394 22609 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:02.068006 21266 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:02.033874 21259 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:02.050228 22656 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251215 08:59:02.063230 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 50327.4us
I20251215 08:59:02.050290 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:02.050473 22664 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.051421 21406 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:02.050323 21407 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:02.054687 22624 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:01.966596 22606 consensus_queue.cc:1059] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20251215 08:59:02.061821 22575 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.062228 22473 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.063309 22514 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.012825 22635 raft_consensus.cc:3055] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.049736 21265 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:02.068529 21307 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.074180 22228 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.165825 21265 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:02.060765 22603 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.093420 21266 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:02.058475 21229 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:02.082507 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 56370.5us
I20251215 08:59:02.082422 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:02.170526 21414 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:59:02.171495 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 103029us
I20251215 08:59:02.172353 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.173646 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:02.095291 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 68238.2us
I20251215 08:59:02.099720 21259 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:02.046885 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:02.157255 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 213366us
I20251215 08:59:02.180243 21379 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.089996 22569 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.181125 22663 consensus_queue.cc:1059] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:02.182062 22609 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.190574 22609 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.191079 22609 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.061842 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:02.193325 21262 raft_consensus.cc:3055] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.193747 21262 raft_consensus.cc:740] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:02.194654 21262 consensus_queue.cc:260] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.195494 22569 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.195901 21262 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.196082 22569 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.198184 22569 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:02.198969 22651 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.178s user 0.004s sys 0.000s
I20251215 08:59:02.199530 22558 raft_consensus.cc:3055] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.200008 22558 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:02.201107 22558 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.202630 22558 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.204896 22663 raft_consensus.cc:3055] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:02.205413 22663 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:02.206415 22663 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.207849 22663 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:02.208108 22578 consensus_meta.cc:287] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.595s user 0.008s sys 0.000s
I20251215 08:59:02.208596 22578 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.178222 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:02.178969 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:02.048910 22519 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:02.183735 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 125360us
I20251215 08:59:02.074584 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:02.185927 21944 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:02.182579 22575 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:02.218673 21261 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:02.141901 21263 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:02.219350 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.078945 22593 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.171128 22635 raft_consensus.cc:740] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:02.221997 21944 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:02.225566 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:02.229588 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:02.265426 21335 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:59:02.265548 21336 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:02.232381 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:02.182799 22656 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.266883 21339 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.174615 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:02.036765 22630 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.272095 22630 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:02.273219 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 51287.6us
I20251215 08:59:02.273712 21972 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.274189 21944 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:02.278359 21262 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:02.288488 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 69194.1us
I20251215 08:59:02.290536 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:02.291539 21339 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
W20251215 08:59:02.296672 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 75416.6us
I20251215 08:59:02.297776 21232 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:02.261344 22583 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:02.299921 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:02.299894 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 215548us
I20251215 08:59:02.266644 22604 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:02.313557 21337 raft_consensus.cc:2368] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3.
W20251215 08:59:02.322124 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.296586 22640 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.294700 22609 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.260586 22633 consensus_queue.cc:1059] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20251215 08:59:02.260859 22569 consensus_queue.cc:1059] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20251215 08:59:02.297077 22671 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.303131 21408 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:02.237846 22519 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.321743 22550 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.323022 22575 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251215 08:59:02.369196 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 66112.6us
I20251215 08:59:02.372395 21381 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:02.322171 22625 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:02.301064 22626 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
W20251215 08:59:02.378665 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.347585 22654 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:02.363282 22574 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.347082 22640 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.373736 22534 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251215 08:59:02.384912 22640 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:02.390657 22680 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.327106 22669 consensus_queue.cc:1059] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20251215 08:59:02.381770 22625 raft_consensus.cc:697] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:02.370308 22635 consensus_queue.cc:260] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.413800 22635 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.414585 22519 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.415588 22514 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.328783 22583 raft_consensus.cc:697] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:02.391440 22680 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.418066 22664 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.394974 22674 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:02.425529 22648 log.cc:927] Time spent T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.463s user 0.001s sys 0.014s
W20251215 08:59:02.425742 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 151651us
I20251215 08:59:02.426265 22550 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:02.428516 22653 log.cc:927] Time spent T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.496s user 0.001s sys 0.001s
I20251215 08:59:02.429929 21944 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.431730 22550 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.432062 22625 consensus_queue.cc:237] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.432516 22550 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.431747 22674 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:02.440091 22674 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.442219 21972 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:02.445689 22593 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.440320 22583 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.469069 22680 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.453877 22674 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.483968 22674 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:02.314236 21230 leader_election.cc:336] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.799s (SENT)
I20251215 08:59:02.518623 21230 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:02.521627 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 207394us
W20251215 08:59:02.525975 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 221223us
W20251215 08:59:02.542073 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 251429us
I20251215 08:59:02.324638 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:02.554821 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 164995us
W20251215 08:59:02.555397 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 329865us
I20251215 08:59:02.561538 21407 raft_consensus.cc:1240] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
W20251215 08:59:02.562045 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.564100 21381 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:02.565732 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:02.566186 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1354) took 1006 ms. Trace:
I20251215 08:59:02.570274 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:02.570986 21334 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:02.571317 21411 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:02.577025 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1040) took 1018 ms. Trace:
I20251215 08:59:02.566779 21409 rpcz_store.cc:276] 1215 08:59:01.559219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:01.559328 (+ 109us) service_pool.cc:225] Handling call
1215 08:59:01.560690 (+ 1362us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:01.561109 (+ 419us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:01.561153 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:01.561990 (+ 837us) log.cc:844] Serialized 501 byte log entry
1215 08:59:01.850053 (+288063us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:02.161847 (+311794us) log.cc:844] Serialized 14 byte log entry
1215 08:59:02.169874 (+ 8027us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:02.169966 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:02.363161 (+193195us) spinlock_profiling.cc:243] Waited 9.63 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:02.565482 (+202321us) raft_consensus.cc:1685] finished
1215 08:59:02.565676 (+ 194us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:02.565978 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74013,"raft.queue_time_us":16630,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":14572,"spinlock_wait_cycles":22212736,"thread_start_us":287658,"threads_started":1,"wal-append.queue_time_us":341022}
I20251215 08:59:02.577489 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:02.580127 22398 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.580940 21411 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:02.577499 21332 rpcz_store.cc:276] 1215 08:59:01.557925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:01.558066 (+ 141us) service_pool.cc:225] Handling call
1215 08:59:02.570023 (+1011957us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398cb6d58e 00007f398610c437 00007f398610c3c2 00007f398610c2f1 00007f398610c26c 00007f398610b522 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:02.576712 (+ 6689us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1694126848}
I20251215 08:59:02.583101 22398 consensus_queue.cc:1059] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:59:02.593673 21304 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:02.594745 22519 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.595577 22519 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.598380 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:02.599440 21337 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:02.618476 22569 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.619596 22519 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:02.619376 22569 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.624892 22550 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:02.628413 21338 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:02.631011 22604 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:02.631521 22604 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:02.632264 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.638067 21332 raft_consensus.cc:1275] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:02.638242 22635 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:02.638957 22635 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.639698 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:02.640592 21336 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.645512 22664 consensus_queue.cc:1059] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:59:02.646395 22539 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:02.647044 22664 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.647250 22539 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.647692 22664 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.650652 22680 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
W20251215 08:59:02.665705 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 49050.4us
I20251215 08:59:02.649333 22567 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.689122 22550 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:02.690196 22630 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:02.690624 22680 raft_consensus.cc:697] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:02.690966 22630 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.693116 22582 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:02.693545 22582 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:02.693917 22582 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:02.694371 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 62368.7us
I20251215 08:59:02.697095 22625 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.697907 22583 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:02.698951 22516 consensus_queue.cc:1059] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:59:02.700369 22516 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:02.701331 22669 raft_consensus.cc:3055] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.701736 22669 raft_consensus.cc:740] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:02.701900 22618 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.702003 22519 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.702551 22618 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.702898 22519 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.703084 22669 consensus_queue.cc:260] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.704535 22669 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.700579 22674 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.697810 22625 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.692255 22680 consensus_queue.cc:237] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.696939 21379 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.707597 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:02.715708 22634 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.707247 22625 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.716622 22634 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.704962 22618 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.709172 22558 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:02.729704 22635 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.707111 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:02.705149 22654 raft_consensus.cc:3055] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.731972 22630 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.722288 22672 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.733220 22593 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:02.735154 22679 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.744318 21307 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:02.690191 22550 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.705677 22674 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.710078 22516 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.744500 22654 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:02.759320 22684 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:02.748525 22519 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:02.750478 22558 raft_consensus.cc:697] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:02.770161 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 50315.9us
I20251215 08:59:02.752899 22634 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.776669 22620 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:02.777706 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 347765us
I20251215 08:59:02.777499 22620 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.778765 21944 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:02.779839 22620 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.780628 22686 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.780735 21944 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.767117 22539 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.768568 22516 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.773257 21379 leader_election.cc:400] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3.
I20251215 08:59:02.783735 22624 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:02.784024 21973 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.775767 22569 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.782336 22677 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.147s
I20251215 08:59:02.782575 22593 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.802281 22655 raft_consensus.cc:3055] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.814999 22664 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.821705 22663 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:02.822450 22672 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.824687 22635 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:02.826128 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:02.826740 22609 raft_consensus.cc:3055] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.831017 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:02.825212 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 85) took 1304 ms. Trace:
I20251215 08:59:02.833683 22605 raft_consensus.cc:3055] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:02.834059 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.804368 21379 leader_election.cc:403] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:02.783727 22674 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:02.840829 21336 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:02.816826 22624 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:02.816074 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:02.774235 22567 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.801750 22679 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.843216 22609 raft_consensus.cc:740] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:02.844832 21408 rpcz_store.cc:276] 1215 08:59:01.517832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:01.525053 (+ 7221us) service_pool.cc:225] Handling call
1215 08:59:01.559903 (+ 34850us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:02.371299 (+811396us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:02.821340 (+450041us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54000d16d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:02.821751 (+ 411us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1504191360}
I20251215 08:59:02.861280 22609 consensus_queue.cc:260] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.849491 22624 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.844838 22655 raft_consensus.cc:740] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:02.856758 22593 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.857475 22682 raft_consensus.cc:2764] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251215 08:59:02.862862 22686 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.866680 22550 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:02.866917 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:02.867333 21972 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:02.859735 22679 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.859771 22514 raft_consensus.cc:3055] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:02.868124 22681 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20251215 08:59:02.868608 22514 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:02.834774 22654 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.870481 22654 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.870079 22514 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.871299 22514 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.784837 22684 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.851706 22620 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:02.873984 22620 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:02.874435 22620 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.855557 21978 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:02.846172 22605 raft_consensus.cc:740] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:02.850411 21341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.002s sys 0.012s
I20251215 08:59:02.781829 22558 consensus_queue.cc:237] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:02.877379 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 96642.1us
W20251215 08:59:02.872556 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 106783us
I20251215 08:59:02.872009 22672 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:02.878698 21335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.037s sys 0.055s
I20251215 08:59:02.889343 22228 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.888932 22605 consensus_queue.cc:260] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.889189 22635 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.888738 21408 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:02.900913 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:02.902195 22473 consensus_meta.cc:287] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.721s user 0.007s sys 0.011s
I20251215 08:59:02.902671 22609 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.901209 21164 catalog_manager.cc:5654] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 2 to 1, leader changed from <none> to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 1 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:02.899457 21341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s user 0.003s sys 0.012s
I20251215 08:59:02.904588 22691 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:02.903437 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 100254us
I20251215 08:59:02.903805 22605 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.868620 22686 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:02.905359 21335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.037s sys 0.055s
W20251215 08:59:02.915557 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.921875 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:02.923787 21261 raft_consensus.cc:2410] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251215 08:59:02.933755 22672 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.935510 22696 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:02.935751 22672 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.936221 22696 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.937600 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:02.938820 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 59829.5us
I20251215 08:59:02.918900 21408 raft_consensus.cc:606] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:02.915763 22691 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.931537 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:02.944337 21230 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:02.948752 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:02.949693 21260 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:02.950062 22696 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:02.951797 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:02.951925 22672 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:02.918269 22473 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.953991 22635 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:02.938938 22655 consensus_queue.cc:260] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:02.960399 21379 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:02.963583 22672 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:02.963987 21379 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:02.965124 22606 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:02.965780 22606 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:02.942626 21408 raft_consensus.cc:614] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:02.973764 21232 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:02.964087 22536 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20251215 08:59:02.977802 21265 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.980907 22575 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:02.981361 22569 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:02.960487 22655 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:02.981953 22569 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.985424 22228 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3.
I20251215 08:59:02.987380 22680 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.987990 22680 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.983156 22681 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:02.966814 22606 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:02.998416 21336 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251215 08:59:02.972890 22686 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:03.003057 22569 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.003140 22680 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.004509 21408 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:03.000375 21336 pending_rounds.cc:85] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 1
I20251215 08:59:03.005121 21336 pending_rounds.cc:107] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251215 08:59:03.005147 21408 raft_consensus.cc:606] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:03.005481 21336 raft_consensus.cc:2889] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:03.005695 21408 raft_consensus.cc:614] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:03.004288 22582 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:02.986666 22669 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:03.006911 21161 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36424 (request call id 223) took 2499 ms. Trace:
I20251215 08:59:03.006611 21164 catalog_manager.cc:5654] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:03.009068 22582 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:03.010007 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 78109.4us
I20251215 08:59:02.998420 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.013150 21341 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.010026 22516 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.014218 22516 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.014605 22516 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.017566 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:03.019652 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 114014us
I20251215 08:59:02.998925 22677 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:03.022639 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 48984.6us
W20251215 08:59:03.023633 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 47853.1us
I20251215 08:59:03.014436 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.024528 21340 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.031976 21265 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:02.989750 22681 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.026641 22677 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.045329 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.020828 22686 raft_consensus.cc:697] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:03.050977 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.054913 22228 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.044070 22672 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.050211 21944 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:03.055475 22536 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.058004 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.058175 22473 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.060665 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.058619 22550 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:03.058842 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.067989 22536 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.070243 22536 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.071115 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.071786 21266 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.073114 22684 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.073755 22691 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:03.074883 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.075995 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.076696 21258 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.077944 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.078631 21259 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.078891 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.067109 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.079842 21262 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.080075 21338 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.046818 22677 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:03.054553 21230 leader_election.cc:336] T b3680af750ef47df88bf5be33eb18fc7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.879s (SENT)
I20251215 08:59:03.034323 22582 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:03.050876 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.065681 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.065879 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.007270 21161 rpcz_store.cc:276] 1215 08:59:00.507179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:00.507720 (+ 541us) service_pool.cc:225] Handling call
1215 08:59:03.006708 (+2498988us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:01.894410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:02.032615 (+138205us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:02.034836 (+ 2221us) write_op.cc:435] Acquired schema lock
1215 08:59:02.037165 (+ 2329us) tablet.cc:660] Decoding operations
1215 08:59:02.050726 (+ 13561us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:02.050791 (+ 65us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:02.050821 (+ 30us) tablet.cc:683] Acquiring locks for 3 operations
1215 08:59:02.051134 (+ 313us) tablet.cc:699] Row locks acquired
1215 08:59:02.051164 (+ 30us) write_op.cc:260] PREPARE: finished
1215 08:59:02.051298 (+ 134us) write_op.cc:270] Start()
1215 08:59:02.051410 (+ 112us) write_op.cc:276] Timestamp: P: 1765789142051254 usec, L: 0
1215 08:59:02.051430 (+ 20us) op_driver.cc:348] REPLICATION: starting
1215 08:59:02.053768 (+ 2338us) log.cc:844] Serialized 1375 byte log entry
1215 08:59:02.474977 (+421209us) op_driver.cc:464] REPLICATION: finished
1215 08:59:02.490791 (+ 15814us) write_op.cc:301] APPLY: starting
1215 08:59:02.490912 (+ 121us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:02.491091 (+ 179us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:02.491282 (+ 191us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:02.491867 (+ 585us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:02.492102 (+ 235us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1215 08:59:02.492172 (+ 70us) write_op.cc:312] APPLY: finished
1215 08:59:02.492457 (+ 285us) log.cc:844] Serialized 35 byte log entry
1215 08:59:02.492648 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:02.492770 (+ 122us) write_op.cc:454] Released schema lock
1215 08:59:02.492819 (+ 49us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:02.493796 (+ 977us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:02.493836 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":66268,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":15517,"apply.run_cpu_time_us":2460,"apply.run_wall_time_us":2720,"num_ops":3,"prepare.queue_time_us":800083,"prepare.run_cpu_time_us":56896,"prepare.run_wall_time_us":413769,"raft.queue_time_us":89726,"raft.run_cpu_time_us":2886,"raft.run_wall_time_us":18852,"replication_time_us":423399,"spinlock_wait_cycles":562048,"thread_start_us":988583,"threads_started":4,"wal-append.queue_time_us":271681}]]}
I20251215 08:59:03.066967 22518 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:03.068084 22535 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:03.095950 22656 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:03.099460 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.102967 22677 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.106016 22677 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:03.107466 21973 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:03.107275 22518 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.107785 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.108132 22677 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:03.109614 22687 raft_consensus.cc:2764] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20251215 08:59:03.109817 22518 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.101079 22228 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.085752 22684 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.119652 22228 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.110715 22535 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.122418 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.124132 22535 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.124971 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.125702 21971 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:03.126753 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:03.127497 21978 raft_consensus.cc:2393] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:03.128337 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.128954 21980 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
W20251215 08:59:03.129695 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.092073 22567 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.093434 22672 raft_consensus.cc:697] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:03.134254 21259 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:03.170149 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.170909 21259 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.171757 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.172662 21259 raft_consensus.cc:2410] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:03.135439 22684 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.173584 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.173944 21266 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.174707 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.174847 22684 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.102978 22656 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.139449 21409 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:03.103562 22663 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.174799 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.141353 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:03.142494 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 89664.6us
I20251215 08:59:03.118495 22582 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.143637 22516 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.144692 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.150485 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.153816 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.154633 22686 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.156908 22681 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:03.157903 22661 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.159240 21381 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:03.161096 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.161286 22620 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:03.162727 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 123845us
I20251215 08:59:03.098937 22674 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:03.134917 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.136040 21229 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:03.193941 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.188726 21263 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.181702 22567 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.196671 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.202149 21381 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:03.202821 21230 leader_election.cc:336] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.197s (SENT)
I20251215 08:59:03.204174 21258 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:03.206842 21977 raft_consensus.cc:1240] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:03.207219 21381 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:03.211591 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 75724.5us
I20251215 08:59:03.216745 21263 raft_consensus.cc:1240] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:03.218761 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.215643 22536 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.222798 21229 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:03.224776 21381 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.225118 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.220146 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.226305 21260 raft_consensus.cc:2393] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:03.225867 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.225860 21414 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:03.229163 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.170136 22672 consensus_queue.cc:237] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.211053 22518 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.233853 21258 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:03.234413 21258 raft_consensus.cc:606] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:03.200971 22681 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20251215 08:59:03.216130 22691 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:03.234750 21258 raft_consensus.cc:614] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251215 08:59:03.221966 22604 consensus_meta.cc:287] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.589s user 0.015s sys 0.000s
I20251215 08:59:03.235329 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.242736 21407 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251215 08:59:03.236996 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.237774 21381 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.231686 22625 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:03.238519 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:03.238512 22660 raft_consensus.cc:2804] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.220275 22536 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.239902 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.240202 21409 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:03.203934 22661 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.211300 22674 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.252023 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.239527 22691 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:03.255149 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.254036 21407 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:03.246465 22625 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.234208 22518 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.252066 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.258910 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:03.249959 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 49898.9us
W20251215 08:59:03.257192 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 47310.6us
I20251215 08:59:03.256999 21266 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.258131 21409 raft_consensus.cc:1240] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:03.214427 22567 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.252137 22536 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.260494 22563 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.246841 22604 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.264751 21944 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:03.257562 22691 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:03.263525 22578 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.271068 22578 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.271143 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.256989 22660 raft_consensus.cc:697] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:03.272174 21407 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:03.272749 21407 raft_consensus.cc:606] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:03.269176 21415 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:03.269691 21410 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:03.264770 22563 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:03.273912 22563 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:03.266080 22702 log.cc:927] Time spent T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
I20251215 08:59:03.274375 22661 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.275085 22620 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.275985 22674 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.259174 22625 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:03.277518 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.278438 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.261225 22518 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.279675 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.281468 22563 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.283432 22563 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.283360 21407 raft_consensus.cc:2393] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:03.285385 21381 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.271518 22578 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.289175 22674 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.289583 22674 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.289927 22674 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.292116 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.292750 21263 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:03.292179 22605 consensus_queue.cc:1059] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20251215 08:59:03.295322 22514 raft_consensus.cc:3055] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:03.295745 22514 raft_consensus.cc:740] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:03.272459 22660 consensus_queue.cc:237] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.296666 22514 consensus_queue.cc:260] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.297942 22514 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:03.298179 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.299424 22674 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.299871 22519 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.300285 22519 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.300606 22519 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.301331 22674 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.304368 21265 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.307705 22620 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251215 08:59:03.311038 22711 log.cc:927] Time spent T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20251215 08:59:03.311746 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.266906 22624 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.314586 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:03.315229 21977 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.284144 22606 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.315951 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.316097 21414 raft_consensus.cc:1240] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
W20251215 08:59:03.279758 22703 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
I20251215 08:59:03.321619 21258 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:03.338143 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.338246 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.339051 21263 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.340265 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.323590 21259 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:03.342193 21259 raft_consensus.cc:606] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Received request to transfer leadership to TS 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:03.342623 21259 raft_consensus.cc:614] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:03.314841 22624 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:03.346884 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:03.347302 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.320792 22516 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:03.351482 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 86824.6us
I20251215 08:59:03.352605 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.352789 21944 leader_election.cc:400] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:03.338433 22582 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.353240 21334 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.355034 22536 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.348369 22656 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:03.355611 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.361896 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20251215 08:59:03.362629 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.364045 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:03.361919 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.356771 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.356729 21944 leader_election.cc:403] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251215 08:59:03.358376 21381 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.364852 21261 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:03.436779 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.361280 21977 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:03.410466 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:03.348604 22674 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.456141 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:03.419014 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.371002 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.357169 21262 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:03.454566 21265 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:03.408792 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.398746 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:03.412698 22654 consensus_meta.cc:287] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.542s user 0.010s sys 0.000s
W20251215 08:59:03.457378 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 99103.6us
I20251215 08:59:03.462952 21978 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.356724 22677 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:03.408255 22604 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.400848 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.325562 22691 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.374392 22536 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.392740 22655 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.360992 21406 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:03.370316 22606 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.366026 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.385939 22567 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.468750 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.468586 21229 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:03.467514 22656 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.471577 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:03.460567 22563 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.472643 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.473851 22625 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.486259 21980 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:03.474545 22519 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.475184 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.384028 22605 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20251215 08:59:03.364935 22518 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:03.480669 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 128027us
I20251215 08:59:03.467501 22674 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.471303 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.466866 22686 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
W20251215 08:59:03.481259 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 110400us
I20251215 08:59:03.498059 21265 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.499176 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.485678 21264 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.487057 22516 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.487545 22563 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.503341 21414 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:03.501341 21257 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.510426 21415 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:03.484093 22677 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.512640 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:03.515640 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:03.504940 22563 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.504495 22516 raft_consensus.cc:697] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:03.499658 22655 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
W20251215 08:59:03.535763 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.517575 22656 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:03.521260 22578 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.535997 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.529708 22654 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251215 08:59:03.538990 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:03.555332 21265 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.555534 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.557518 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:03.557591 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.560364 21407 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:03.560454 21260 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.557976 21413 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:03.557579 22516 consensus_queue.cc:237] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.560913 21381 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.561846 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.550015 22691 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.562846 21265 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.564478 21409 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:03.552974 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:03.561650 21406 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.565886 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.566090 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:03.568173 21977 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:03.572722 22677 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.546985 22656 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:03.574427 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.577791 21258 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:03.575073 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:03.579955 21413 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:03.579454 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:03.581581 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.582288 21336 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:59:03.585996 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.587172 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.587309 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.588016 21340 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.588299 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.588451 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:03.588697 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.567716 22661 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.588980 21333 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.589166 22681 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:03.590049 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.590489 22681 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.590780 21336 raft_consensus.cc:2393] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:03.591140 22681 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:03.591862 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.592439 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:03.589305 22661 raft_consensus.cc:697] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:03.593969 22661 consensus_queue.cc:237] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.594816 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.570928 22654 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.581071 22605 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:03.596449 22605 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.581638 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:03.571017 22519 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.598418 22605 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:03.579946 21974 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:03.597553 22656 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.598906 22605 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:03.574445 22518 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.602380 22578 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.565279 22625 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.604441 22620 raft_consensus.cc:2764] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251215 08:59:03.604513 21257 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.595155 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.605607 21336 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:03.605711 22714 log.cc:927] Time spent T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20251215 08:59:03.580417 22674 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.613639 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.614069 21406 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:03.606925 22620 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:03.615881 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:03.608568 21973 raft_consensus.cc:2368] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3.
I20251215 08:59:03.609387 21946 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:03.594859 21307 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:03.642365 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 47601.2us
I20251215 08:59:03.642836 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:03.643343 21340 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.607942 21381 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.580871 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:03.644711 21414 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:03.618798 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:03.645697 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:03.647483 21232 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:03.649245 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:03.649958 21336 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.626305 22655 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.627565 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:03.629266 22656 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.630071 22578 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.630908 22674 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:03.633749 22677 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.635142 22558 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:03.637277 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:03.638374 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.639173 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:03.672667 21333 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.649175 22609 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.624704 21974 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:03.621390 21978 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:03.620991 21261 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.652752 21266 raft_consensus.cc:1275] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:03.655436 21232 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:03.678128 21381 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.679520 21379 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.679629 22398 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:03.676548 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.656119 21260 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.658232 21338 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:03.658825 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 49511.9us
I20251215 08:59:03.665721 21265 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.645581 21257 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.651132 22518 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.666209 22655 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.669835 22686 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:03.686472 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.673146 21971 raft_consensus.cc:3060] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.688881 21229 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:03.685732 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:03.691658 21260 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:03.691919 21335 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.650046 22563 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.692900 21379 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:03.674979 22519 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:03.674680 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:03.696167 21381 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.659205 22605 consensus_queue.cc:1059] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251215 08:59:03.697647 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:03.698356 21379 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.699497 21337 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.697125 21332 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:03.699005 22398 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.702265 21946 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:03.700766 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:03.704603 22398 raft_consensus.cc:697] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:03.705255 22228 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:03.705890 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.689620 22686 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.694128 22660 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.673969 22656 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.709332 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:03.694977 22518 raft_consensus.cc:697] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:03.703729 22536 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251215 08:59:03.722955 22398 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.723935 21264 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.724103 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.724732 22426 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.695580 22535 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.695470 22654 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.684785 22514 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251215 08:59:03.679668 22578 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.696444 22519 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.750654 21414 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:03.720860 21336 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.695868 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.703838 21334 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:59:03.703608 22624 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:03.713884 22663 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20251215 08:59:03.732281 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:03.683033 22674 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:03.733171 22563 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.707219 22620 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:03.730021 22582 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.715320 22660 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:03.738374 22656 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:03.739248 22518 consensus_queue.cc:237] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.740348 21340 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.732733 22536 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:03.771584 22624 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:03.744289 21333 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.690821 22677 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.712107 22684 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:03.712379 22686 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:03.760608 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54753.5us
I20251215 08:59:03.781831 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.802278 21973 raft_consensus.cc:2393] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:03.775269 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:03.776768 22228 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.747272 22625 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251215 08:59:03.788021 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 80293.2us
I20251215 08:59:03.763037 22519 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:03.775600 22609 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:03.804720 21381 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:03.770934 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 61728.1us
I20251215 08:59:03.788221 22680 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:03.814205 21307 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:03.815616 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:03.773319 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.762763 22654 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.793990 22660 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:03.808068 21974 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:03.793784 22620 raft_consensus.cc:697] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:03.821257 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:03.961979 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:03.880358 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 67114us
I20251215 08:59:04.011780 21338 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.082544 21379 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:59:04.082701 21230 leader_election.cc:336] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.506s (SENT)
I20251215 08:59:04.066347 21335 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.083516 21230 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:04.002106 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.803676 22624 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:03.871469 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 66825.8us
I20251215 08:59:03.881439 22228 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:04.085359 22228 raft_consensus.cc:697] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.078454 21337 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
W20251215 08:59:03.992607 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:04.090472 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 274952us
I20251215 08:59:04.091184 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:04.083856 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:04.091885 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.871634 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:03.987145 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:03.875718 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:04.095293 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.097738 21266 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:04.098457 21229 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:04.092015 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:03.890074 22680 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.110076 22680 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.858155 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:04.112363 22680 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.945339 21262 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251215 08:59:04.112790 21264 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.107702 22620 consensus_queue.cc:237] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:03.805997 22536 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:03.872089 22656 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251215 08:59:03.881017 22725 log.cc:927] Time spent T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20251215 08:59:03.880205 22609 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.124683 22654 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:04.125685 22609 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:03.948683 22535 raft_consensus.cc:697] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.127329 22535 consensus_queue.cc:237] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.128633 21264 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:03.822692 22684 raft_consensus.cc:697] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.095124 22563 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:04.129987 22563 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:04.129911 22684 consensus_queue.cc:237] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.134691 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.130378 22563 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:04.037178 22724 log.cc:927] Time spent T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
W20251215 08:59:04.138118 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54323.7us
W20251215 08:59:04.143432 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.146103 21261 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251215 08:59:04.146598 21260 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:59:04.149312 21379 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:03.984284 22677 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:04.146970 22228 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.151079 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:04.089655 22516 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:04.081185 22674 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.969555 22578 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:03.979347 22655 raft_consensus.cc:3055] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:04.051707 21971 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.150978 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:04.155723 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:04.162310 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 79639.9us
I20251215 08:59:04.157186 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.163549 21338 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.120930 22536 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:04.140550 22691 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:04.154075 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:04.138427 22426 raft_consensus.cc:697] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.146377 22514 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.147543 22680 raft_consensus.cc:2764] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251215 08:59:04.149351 22656 raft_consensus.cc:2764] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251215 08:59:04.152115 22731 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.158712 22672 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.154429 22654 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:04.154155 22606 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:04.150293 21257 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.188572 21338 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.069872 22686 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.191270 21339 raft_consensus.cc:1275] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:04.159840 22677 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:04.159972 22655 raft_consensus.cc:740] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:04.195292 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.177457 22516 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:04.185052 22582 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:04.210860 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.185622 21974 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:04.189044 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:04.193027 22691 raft_consensus.cc:697] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:04.216446 21336 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:04.212309 22654 raft_consensus.cc:697] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:04.218537 21338 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.196475 22606 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:04.196151 22672 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.190284 22514 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.224308 21304 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:04.224022 22606 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:04.223343 21378 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.68:45831 timed out after 2.029s (SENT) [suppressed 22 similar messages]
I20251215 08:59:04.219374 21408 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:04.194085 22731 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.214440 22680 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.247850 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:04.249434 22654 consensus_queue.cc:237] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.250136 22686 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:04.250635 22656 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:04.253161 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 62291.4us
I20251215 08:59:04.210026 22677 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:04.288486 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 196481us
W20251215 08:59:04.287752 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 58714.9us
I20251215 08:59:04.292953 21338 raft_consensus.cc:1240] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 3. Ops: []
I20251215 08:59:04.293023 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.251181 22661 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:04.308111 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:04.309140 21414 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:04.281872 22582 raft_consensus.cc:697] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.327243 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.311371 21980 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:04.332267 21410 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:04.333137 21338 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251215 08:59:04.334617 22731 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:04.334362 22582 consensus_queue.cc:237] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.334987 21304 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:04.335562 22741 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20251215 08:59:04.336097 22609 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:04.336617 22609 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:04.251713 22426 consensus_queue.cc:237] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.336558 22691 consensus_queue.cc:237] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.337179 22609 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:04.340802 22677 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:04.290374 22656 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.344305 22656 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:04.345646 22606 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.347700 22606 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:04.299814 22655 consensus_queue.cc:260] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.350463 21339 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:04.351451 22660 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.354746 22609 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.353849 22672 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:04.366019 21415 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:04.352926 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:04.377921 21411 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:04.382450 22660 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:04.382731 22723 log.cc:927] Time spent T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.595s user 0.004s sys 0.000s
W20251215 08:59:04.383399 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.323697 22738 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:04.393765 21379 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:04.299789 22680 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.385568 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:04.465361 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 72350us
I20251215 08:59:04.469874 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:04.466746 21332 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 3. Ops: []
I20251215 08:59:04.441991 22742 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:04.464175 22519 consensus_meta.cc:287] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.580s user 0.005s sys 0.005s
I20251215 08:59:04.314814 22661 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.475711 21973 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:04.476331 22514 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:04.389153 22732 consensus_queue.cc:1059] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.089s
I20251215 08:59:04.387898 22563 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.477972 21408 raft_consensus.cc:2368] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3.
I20251215 08:59:04.464663 22655 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.464907 22738 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:04.485478 21340 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:04.497088 21337 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:04.479553 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:04.473716 22660 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:04.507251 22514 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20251215 08:59:04.499505 22742 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.513968 22606 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20251215 08:59:04.512326 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.523254 21332 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:04.513161 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:04.524034 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.524163 21339 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:04.476790 22737 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251215 08:59:04.515312 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.494771 22733 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251215 08:59:04.525894 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.523381 22742 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:04.527341 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:04.528283 22680 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:04.528453 22757 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.053s user 0.004s sys 0.005s
I20251215 08:59:04.502367 22732 consensus_queue.cc:1243] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer 195a56ac72a84018880bfb3bf6e6ef1d log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251215 08:59:04.545354 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.546170 21336 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.531373 21341 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:04.529181 21409 raft_consensus.cc:1240] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251215 08:59:04.513762 22738 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.498883 22519 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:04.556247 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.529079 21410 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:04.532351 22733 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:04.568611 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.529489 22730 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.570298 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:04.570446 22730 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.570961 21980 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:04.574828 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 492711us
I20251215 08:59:04.545923 22660 raft_consensus.cc:697] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.578162 21338 raft_consensus.cc:1240] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:04.586019 21336 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.580170 21259 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:04.559214 22609 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:04.583501 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 273868us
W20251215 08:59:04.589409 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.568359 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:04.597628 21332 raft_consensus.cc:1240] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:04.572810 22514 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:04.590684 21379 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:04.594364 21232 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:04.594553 21381 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:04.574231 22733 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:04.572640 22567 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:04.581503 22516 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:04.568756 22536 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251215 08:59:04.610882 21381 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:04.598135 22732 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20251215 08:59:04.625085 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.625263 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:04.598771 22625 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:04.631284 21341 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:04.609558 21406 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20251215 08:59:04.654248 21381 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:04.610139 22737 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:04.678462 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.715555 21266 raft_consensus.cc:1240] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:04.665352 22756 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.613430 22567 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.607314 21976 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:04.679921 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.725680 21341 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:04.667114 21980 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:04.629294 22730 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:04.629761 22516 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.626670 21411 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:04.729190 22516 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:04.729986 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.627161 22661 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:04.612229 22426 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:04.734598 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:04.734983 21304 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:04.735160 21266 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.677059 22736 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:04.633925 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.659760 22578 raft_consensus.cc:993] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:59:04.731137 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.626312 22519 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:04.733212 21341 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:04.746443 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.747862 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.736048 21332 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:04.723675 22536 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:04.678365 21415 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:04.722597 22625 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.620956 22760 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:04.738467 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:04.739069 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.754767 21263 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
W20251215 08:59:04.628044 22624 consensus_meta.cc:287] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.512s user 0.009s sys 0.000s
I20251215 08:59:04.628638 22563 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:04.733079 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 79370.9us
I20251215 08:59:04.610199 21972 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:04.749231 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.749094 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.750195 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:04.760721 22398 consensus_queue.cc:1059] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:04.659933 22606 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.090s
I20251215 08:59:04.613229 22514 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.734862 22426 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:04.738025 22736 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:04.756995 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.662535 22761 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20251215 08:59:04.760550 21332 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:04.774602 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:04.774927 21265 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:04.775506 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.775811 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:04.777158 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:04.732120 22605 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:04.765228 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.783354 21337 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:04.761967 22536 raft_consensus.cc:697] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.738767 22686 consensus_queue.cc:1059] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20251215 08:59:04.770100 22606 raft_consensus.cc:1081] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Signalling peer f0824fa5450345719cac22fc6ae3ce2e to start an election
I20251215 08:59:04.770905 22680 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.737620 22609 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:04.742017 22516 consensus_queue.cc:1059] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251215 08:59:04.777712 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.794114 22398 raft_consensus.cc:3055] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:04.794540 22398 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:04.777676 22677 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:04.794307 21332 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:04.781097 22514 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:04.773449 22656 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.777868 22742 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:04.778088 22563 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:04.742017 22730 consensus_queue.cc:1059] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20251215 08:59:04.723775 22660 consensus_queue.cc:237] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.758427 22760 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:04.776479 22426 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:04.781741 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.808323 21338 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:04.776655 22736 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.798290 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.809574 21332 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:04.799785 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.750078 22756 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.799672 22681 raft_consensus.cc:3055] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:04.815874 21266 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:04.803493 21232 leader_election.cc:400] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3.
I20251215 08:59:04.828778 21232 leader_election.cc:403] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20251215 08:59:04.829484 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 69390.2us
I20251215 08:59:04.759474 22624 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.808727 22677 raft_consensus.cc:2764] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251215 08:59:04.853825 22514 consensus_queue.cc:1059] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:04.855022 22514 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:04.858093 22624 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:04.858888 21412 raft_consensus.cc:1240] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:04.861637 21410 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:04.830749 22516 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3.
I20251215 08:59:04.824265 22563 raft_consensus.cc:697] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:04.863044 21332 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:04.864049 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.865384 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.866420 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.867043 21332 raft_consensus.cc:2368] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3.
I20251215 08:59:04.866986 21232 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:04.867298 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:04.797327 22605 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.867811 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.811379 22742 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:04.809226 22760 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.833772 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.872813 22426 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:04.873682 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.903261 21332 raft_consensus.cc:2376] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2. Re-sending same reply.
I20251215 08:59:04.801066 22656 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:04.907239 21261 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.012s sys 0.029s
I20251215 08:59:04.907598 22426 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:04.907929 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:04.907634 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:04.910580 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.828048 22681 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:04.808449 22680 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.909160 21415 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:04.911413 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:04.907790 21261 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.012s sys 0.029s
I20251215 08:59:04.910663 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.903066 22516 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:04.833016 22625 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:04.932320 21335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.000s sys 0.045s
I20251215 08:59:04.939468 21266 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251215 08:59:04.939862 21335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s user 0.000s sys 0.045s
I20251215 08:59:04.940923 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:04.911594 22582 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.050s sys 0.059s
I20251215 08:59:04.954562 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.955277 21337 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:04.955574 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.955876 21335 raft_consensus.cc:2376] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2. Re-sending same reply.
I20251215 08:59:04.956454 21332 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:04.959689 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:04.958305 21266 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:04.863986 22563 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:04.939000 22516 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:04.947425 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:04.934186 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.963616 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:04.911221 22742 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:04.969902 21332 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:04.871703 22536 consensus_queue.cc:237] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:04.950416 22582 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s user 0.050s sys 0.059s
I20251215 08:59:04.970072 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:05.025563 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 56316.2us
I20251215 08:59:05.002398 21338 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:05.025740 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:05.029776 22691 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.010s sys 0.016s
W20251215 08:59:05.030164 22691 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.010s sys 0.016s
I20251215 08:59:05.033326 21333 raft_consensus.cc:1240] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:05.039824 21333 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:05.043021 21339 raft_consensus.cc:1275] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.047385 22779 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20251215 08:59:05.064323 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 372966us
I20251215 08:59:05.023005 21335 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:05.037540 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 306488us
I20251215 08:59:05.021344 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.091389 21414 raft_consensus.cc:1240] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:05.022266 22680 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:05.100023 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:05.095014 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.118269 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:04.969388 22516 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.001974 22742 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:05.084532 21334 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.001s sys 0.031s
W20251215 08:59:05.085151 21973 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.009s sys 0.015s
I20251215 08:59:04.977511 22756 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.137460 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:05.139966 22426 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:05.001838 22739 raft_consensus.cc:3055] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
W20251215 08:59:05.142098 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 47195.1us
W20251215 08:59:05.125492 21334 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.799s user 0.001s sys 0.032s
I20251215 08:59:05.119939 22609 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:05.151292 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.151309 21378 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:04.999065 22729 raft_consensus.cc:3055] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
W20251215 08:59:05.126873 21973 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.010s sys 0.017s
I20251215 08:59:05.146867 21332 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:05.121965 22737 raft_consensus.cc:697] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:05.150381 21381 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:05.162189 21379 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:05.165745 21332 raft_consensus.cc:1240] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:05.169386 21336 raft_consensus.cc:1275] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.169930 21332 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:05.173951 21332 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:05.141665 22672 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:05.217049 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.142694 22739 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:05.153362 22605 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.129660 22736 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.220687 22654 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:05.162730 22656 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.218724 22732 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:05.251281 22609 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.222179 22750 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:04.964478 22681 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.250204 22654 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.251591 22732 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.171825 22751 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:05.216331 22733 consensus_meta.cc:287] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.582s user 0.010s sys 0.000s
I20251215 08:59:05.263935 21264 raft_consensus.cc:1275] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.265100 22736 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:05.171198 22729 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:05.256901 21341 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:05.221361 22624 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:05.252835 22687 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:05.267869 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:05.268944 21232 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:05.246457 22578 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
W20251215 08:59:05.270705 22228 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.007s sys 0.040s
W20251215 08:59:05.265712 22764 log.cc:927] Time spent T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
W20251215 08:59:05.302357 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:05.263943 22733 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.272677 21335 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:05.310428 21332 raft_consensus.cc:1240] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
W20251215 08:59:05.306344 22228 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.572s user 0.007s sys 0.044s
I20251215 08:59:05.259616 22732 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.265120 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.314237 21335 raft_consensus.cc:1275] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.279260 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1226) took 1112 ms. Trace:
I20251215 08:59:05.309022 22681 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:05.330924 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 180637us
I20251215 08:59:05.307684 22578 raft_consensus.cc:697] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:05.336190 21381 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:05.319284 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 99) took 1166 ms. Trace:
W20251215 08:59:05.354585 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 203383us
I20251215 08:59:05.321952 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.325659 22768 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:05.274518 22751 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.329964 22654 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:05.322656 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 85) took 1969 ms. Trace:
I20251215 08:59:05.380667 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 91) took 1217 ms. Trace:
I20251215 08:59:05.323762 22762 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:05.356510 21381 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:05.375485 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:05.377075 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 204049us
I20251215 08:59:05.375241 22768 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:05.354331 22660 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:05.381222 21260 rpcz_store.cc:276] 1215 08:59:04.163129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.163232 (+ 103us) service_pool.cc:225] Handling call
1215 08:59:04.186335 (+ 23103us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:04.199221 (+ 12886us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:04.199264 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:04.214049 (+ 14785us) log.cc:844] Serialized 26 byte log entry
1215 08:59:04.389828 (+175779us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b48000027d8. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398cacde21 00007f398cac4eb7 00007f398cac4d4a 00007f398cbbbf7f 00007f398cbbbc47 00007f398cbd026c 00007f398cbd0218 00007f398cbd0198 00007f398cbd0153 00007f398cbcf3fc 00007f398bb0a516
1215 08:59:04.572459 (+182631us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:04.572502 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:04.573827 (+ 1325us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:05.375884 (+802057us) raft_consensus.cc:1685] finished
1215 08:59:05.375917 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:05.380583 (+ 4666us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1763,"raft.queue_time_us":1946,"raft.run_cpu_time_us":2359,"raft.run_wall_time_us":196414,"spinlock_wait_cycles":9560448,"thread_start_us":330642,"threads_started":1,"wal-append.queue_time_us":707120}
I20251215 08:59:05.387564 22768 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:05.388485 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.306913 22687 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.390515 21260 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:59:05.310631 22737 consensus_queue.cc:237] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:05.386723 22744 log.cc:927] Time spent T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.778s user 0.008s sys 0.005s
I20251215 08:59:05.399869 21379 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:05.387624 22660 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:05.402626 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:05.403825 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.406145 21260 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.327329 21334 rpcz_store.cc:276] 1215 08:59:04.157711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.158703 (+ 992us) service_pool.cc:225] Handling call
1215 08:59:04.205533 (+ 46830us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:04.228781 (+ 23248us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:04.233293 (+ 4512us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:04.244041 (+ 10748us) log.cc:844] Serialized 26 byte log entry
1215 08:59:04.321544 (+ 77503us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:05.167326 (+845782us) log.cc:844] Serialized 14 byte log entry
1215 08:59:05.242024 (+ 74698us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:05.249207 (+ 7183us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:05.255114 (+ 5907us) raft_consensus.cc:1685] finished
1215 08:59:05.255136 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:05.270511 (+ 15375us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40971,"prepare.queue_time_us":152142,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":46327,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":12241,"spinlock_wait_cycles":1792,"thread_start_us":881561,"threads_started":2,"wal-append.queue_time_us":222837}
I20251215 08:59:05.400839 22624 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:05.354030 21406 rpcz_store.cc:276] 1215 08:59:04.152503 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.153544 (+ 1041us) service_pool.cc:225] Handling call
1215 08:59:04.186664 (+ 33120us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:04.690812 (+504148us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:05.304476 (+613664us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54001b7bd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:05.317064 (+ 12588us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":912739584}
I20251215 08:59:05.409734 22660 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.413686 22733 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:05.416550 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.417711 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:05.380005 21262 rpcz_store.cc:276] 1215 08:59:03.353246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:03.476768 (+123522us) service_pool.cc:225] Handling call
1215 08:59:03.748311 (+271543us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:04.112942 (+364631us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:04.664839 (+551897us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54001342d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:05.322440 (+657601us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54001342d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:05.322573 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":864522880}
I20251215 08:59:05.418227 21266 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:05.419091 21262 raft_consensus.cc:1275] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.392043 22730 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.422875 22398 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.335019 22729 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.432175 21266 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:05.398697 22766 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.438484 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 100) took 1285 ms. Trace:
I20251215 08:59:05.428570 22604 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.438892 21412 rpcz_store.cc:276] 1215 08:59:04.152719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.152824 (+ 105us) service_pool.cc:225] Handling call
1215 08:59:04.660372 (+507548us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:04.660661 (+ 289us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:04.859246 (+198585us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:05.438220 (+578974us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54000eb854. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:05.438413 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":482295808}
I20251215 08:59:05.439652 22604 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.433736 21266 raft_consensus.cc:1275] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251215 08:59:05.443490 21266 pending_rounds.cc:85] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 1
I20251215 08:59:05.443941 21266 pending_rounds.cc:107] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251215 08:59:05.335101 22739 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.444474 21266 raft_consensus.cc:2889] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251215 08:59:05.456866 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 53150.4us
W20251215 08:59:05.458415 22795 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251215 08:59:05.458285 22516 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.458976 22684 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251215 08:59:05.459967 22798 log.cc:927] Time spent T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251215 08:59:05.461117 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:05.461300 21381 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:05.461766 21973 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:05.462611 22781 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:05.462782 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:05.463127 22781 raft_consensus.cc:697] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:05.398756 22661 consensus_queue.cc:1048] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20251215 08:59:05.433386 22606 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:05.460893 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:05.470147 22792 log.cc:927] Time spent T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
I20251215 08:59:05.390441 22762 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.471652 21260 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:05.476120 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.476382 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.477474 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.488827 21408 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:05.489368 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.489434 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.479249 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:05.479533 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:05.473420 21980 raft_consensus.cc:2368] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4.
I20251215 08:59:05.490775 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.475613 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.473618 22687 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.491837 21406 raft_consensus.cc:2393] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:05.480671 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.492533 21411 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:05.492652 22779 consensus_queue.cc:1048] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:05.493127 21381 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:05.494210 22739 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:05.481096 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.494978 22604 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.497686 22751 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:05.498353 22738 consensus_meta.cc:287] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.928s user 0.009s sys 0.003s
I20251215 08:59:05.498723 22738 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.499292 21973 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7"
dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:46158
W20251215 08:59:05.499672 21307 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.003s sys 0.001s
I20251215 08:59:05.499936 21973 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:05.500293 21973 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:05.500712 22738 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:05.501108 22736 consensus_queue.cc:1048] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
W20251215 08:59:05.499939 21307 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.003s sys 0.001s
W20251215 08:59:05.502086 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 724938us
I20251215 08:59:05.480011 22768 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:05.491916 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 56704us
I20251215 08:59:05.473704 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:05.482667 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.505156 21340 raft_consensus.cc:2393] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:05.505272 21415 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:05.483139 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.485582 21378 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:05.486708 22567 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:05.486841 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:05.443859 22729 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.487733 22398 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:05.489698 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:05.478394 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.474262 22605 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.511663 22426 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:05.472496 21972 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251215 08:59:05.507679 21979 raft_consensus.cc:2368] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3.
I20251215 08:59:05.508556 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:05.514406 21378 leader_election.cc:400] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Vote denied by peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831) with higher term. Message: Invalid argument: T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4.
I20251215 08:59:05.514724 21378 leader_election.cc:403] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:05.514766 21307 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.520136 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:05.502187 22609 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:05.524901 22789 log.cc:927] Time spent T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20251215 08:59:05.524739 22609 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.524951 22781 consensus_queue.cc:237] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.525694 21264 raft_consensus.cc:1275] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.510106 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:05.527432 22609 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:05.527618 21262 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:05.433247 22578 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.520817 22605 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:05.530841 22760 consensus_meta.cc:287] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.599s user 0.009s sys 0.000s
I20251215 08:59:05.517918 22751 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:05.531198 22605 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:05.531726 22751 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.532640 22516 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.533552 21334 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.533667 22751 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.520736 22656 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.521867 21332 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:05.543681 21378 leader_election.cc:400] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Vote denied by peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831) with higher term. Message: Invalid argument: T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3.
I20251215 08:59:05.512260 22674 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:05.545100 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:05.509572 22606 raft_consensus.cc:697] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:05.545698 21338 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.518256 22779 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:05.553643 22779 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:05.554013 22779 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.450402 22766 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.518678 21408 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 4
I20251215 08:59:05.518551 22738 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:05.537542 22604 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.550434 22762 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.551802 21339 raft_consensus.cc:1275] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.513480 22799 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20251215 08:59:05.531240 22760 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:05.576766 21258 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:05.577481 22760 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:05.577945 22655 raft_consensus.cc:3055] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:05.578343 22655 raft_consensus.cc:740] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:05.564348 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:05.614441 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 50192.1us
I20251215 08:59:05.616145 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:59:05.619408 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 64965.8us
I20251215 08:59:05.621652 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.622385 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.622581 22738 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.623098 21259 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:05.624439 21307 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.630932 21338 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:05.628260 21230 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.635053 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.635493 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.635660 21259 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:05.567651 22609 consensus_queue.cc:1059] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
W20251215 08:59:05.638089 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 77055.8us
I20251215 08:59:05.611145 22654 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:05.632731 21332 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:05.565945 22750 consensus_queue.cc:1048] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20251215 08:59:05.639592 21230 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:05.659013 21971 raft_consensus.cc:2410] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251215 08:59:05.625423 22680 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:05.641237 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.647876 22751 consensus_queue.cc:1059] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:05.659906 21230 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.648047 22663 consensus_queue.cc:1059] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:05.661090 22663 raft_consensus.cc:3055] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:05.661291 21230 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.662047 22514 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.662581 22398 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:05.663045 22398 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:05.663132 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.663504 22398 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.663990 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:05.664433 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.657975 22687 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:05.660550 21265 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:05.661412 22663 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:05.579067 22604 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.670533 21259 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251215 08:59:05.662639 22514 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.650893 22681 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:05.637298 22738 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.625571 22674 raft_consensus.cc:697] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:05.670825 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.677531 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.638705 22768 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.643836 22762 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:05.679687 21414 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:05.675851 22663 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.681059 22398 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.681870 21415 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251215 08:59:05.676630 22604 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.659916 22680 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:05.674623 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:05.691203 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:05.673434 22751 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.700471 22398 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:05.664665 22788 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20251215 08:59:05.686375 21410 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:05.686046 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.698565 22514 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20251215 08:59:05.676867 22660 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.708992 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.709239 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:05.715602 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:05.708041 22663 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:05.642140 22655 consensus_queue.cc:260] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.717253 22674 consensus_queue.cc:237] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.642402 22606 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.700014 22605 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:05.719177 22672 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:05.683369 22762 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:05.719831 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.761268 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.761561 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:05.762267 21261 raft_consensus.cc:2393] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:05.762388 21260 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:05.692492 22680 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:05.725235 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 77547.2us
I20251215 08:59:05.704604 22751 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.721275 22398 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20251215 08:59:05.731884 21264 raft_consensus.cc:1240] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:05.731959 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:05.735311 22567 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.740475 22654 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.775063 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.775331 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:05.748497 21414 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:05.776376 21261 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:05.749248 21407 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:05.756599 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.759615 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.702476 21973 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.739293 22729 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:05.708940 22766 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:05.760349 22672 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.778080 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:05.768849 22762 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:05.775844 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:05.783803 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.769703 22660 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:05.790124 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:05.790743 21414 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:05.790131 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:05.791591 21415 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:05.791915 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:05.793072 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.777715 22609 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.791519 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.794937 21338 raft_consensus.cc:1275] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.803323 21339 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251215 08:59:05.796111 21975 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:05.796495 21411 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:05.797996 21263 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:05.784972 22729 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.760061 22605 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.805685 21414 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:05.810880 21263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1250) took 1090 ms. Trace:
I20251215 08:59:05.812567 21263 rpcz_store.cc:276] 1215 08:59:04.720840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.761443 (+ 40603us) service_pool.cc:225] Handling call
1215 08:59:05.797510 (+1036067us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:05.797678 (+ 168us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:05.810574 (+ 12896us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:05.810811 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9638144}
I20251215 08:59:05.789179 22654 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:05.829743 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.831782 22609 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:05.778927 22567 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.780400 22738 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20251215 08:59:05.800535 21257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1241) took 1094 ms. Trace:
I20251215 08:59:05.835350 22751 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:05.844307 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.848234 21412 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:05.781270 22742 consensus_queue.cc:1059] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
W20251215 08:59:05.866238 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 132956us
W20251215 08:59:05.865742 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 130495us
W20251215 08:59:05.873878 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 98901.5us
I20251215 08:59:05.874675 21307 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:05.875885 21415 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:05.876168 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:05.880357 21414 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:05.900207 21946 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:05.869689 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:05.866084 22624 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20251215 08:59:05.908641 22567 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.908943 21412 raft_consensus.cc:1217] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Deduplicated request from leader. Original: 1.4->[2.5-2.5] Dedup: 2.5->[]
I20251215 08:59:05.909688 21412 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251215 08:59:05.886818 22681 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:05.865119 21257 rpcz_store.cc:276] 1215 08:59:04.706384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.722870 (+ 16486us) service_pool.cc:225] Handling call
1215 08:59:04.748753 (+ 25883us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:04.780377 (+ 31624us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:04.780425 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:04.780483 (+ 58us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:04.780524 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:05.800302 (+1019778us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b540020c454. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:05.800363 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:05.800481 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55201024}
I20251215 08:59:05.891975 22742 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:05.899780 22514 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.869087 22609 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.868880 22779 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.942488 22831 consensus_queue.cc:1059] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:05.947088 22514 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:05.926117 22738 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:05.911983 22762 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.983230 22654 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:05.991436 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 145408us
I20251215 08:59:05.923264 22681 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:05.995147 21409 raft_consensus.cc:2393] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:05.943977 22831 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:06.020921 22823 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.023716 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1204) took 1149 ms. Trace:
I20251215 08:59:05.991199 22732 raft_consensus.cc:3055] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
W20251215 08:59:06.028342 22811 log.cc:927] Time spent T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.403s user 0.003s sys 0.000s
W20251215 08:59:06.028702 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 103318us
I20251215 08:59:06.035161 21260 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:06.036648 21258 rpcz_store.cc:276] 1215 08:59:04.874445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.874593 (+ 148us) service_pool.cc:225] Handling call
1215 08:59:06.023438 (+1148845us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b5400356e54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:06.023608 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1529998208}
I20251215 08:59:06.038213 22737 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:06.003700 21973 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.039448 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 100) took 1264 ms. Trace:
I20251215 08:59:06.015477 22691 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:05.984640 22742 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.038527 22823 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.042711 21337 rpcz_store.cc:276] 1215 08:59:04.771263 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.793697 (+ 22434us) service_pool.cc:225] Handling call
1215 08:59:06.035404 (+1241707us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400153e54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:06.035505 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331749120}
I20251215 08:59:06.039333 22582 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:06.039692 22729 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.016031 22654 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.039974 22732 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:06.039129 22656 raft_consensus.cc:3055] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:06.054812 22731 consensus_queue.cc:1048] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:06.061482 22605 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.063962 22656 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:06.082430 22656 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.083607 22656 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:06.047729 22691 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.079401 22518 consensus_queue.cc:1048] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:06.080044 22660 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:06.066123 22582 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.081275 22779 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.078068 22681 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:06.087191 21232 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:06.092660 21332 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:06.094229 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:06.113108 21258 raft_consensus.cc:1275] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251215 08:59:06.111537 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:06.116989 21339 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:06.132644 21232 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:06.133791 21407 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:06.078095 22823 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.136165 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:06.136358 22398 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.138697 22398 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.059377 22742 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.119575 22738 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:06.146440 22827 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.132543 22681 raft_consensus.cc:3055] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:06.147177 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:06.171919 22738 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.147660 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:06.138192 22604 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:06.150998 22582 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.154647 22672 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.155402 22398 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.137090 22605 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.019s
I20251215 08:59:06.214243 21415 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
W20251215 08:59:06.196971 21340 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1238) took 1686 ms (client timeout 1668 ms). Trace:
W20251215 08:59:06.216146 21340 rpcz_store.cc:269] 1215 08:59:04.510261 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.515014 (+ 4753us) service_pool.cc:225] Handling call
1215 08:59:06.188487 (+1673473us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54001243d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:06.196885 (+ 8398us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2237583744}
I20251215 08:59:06.134778 22760 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:06.130959 22732 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.219365 22732 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:06.126011 22609 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.157728 22654 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.159339 22691 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.109445 22680 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:06.170459 22827 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.174275 22687 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:06.174386 22738 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.176662 22655 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:06.179077 21408 consensus_meta.cc:287] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.600s user 0.014s sys 0.005s
I20251215 08:59:06.179991 22742 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:05.168612 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:06.195538 21406 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:06.234949 22687 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.137434 22762 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.191495 21161 catalog_manager.cc:5654] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:06.208568 22604 raft_consensus.cc:697] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:06.212934 22672 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:06.218241 22815 log.cc:927] Time spent T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.594s user 0.003s sys 0.000s
W20251215 08:59:06.229243 22807 log.cc:927] Time spent T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.600s user 0.000s sys 0.003s
W20251215 08:59:06.229944 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:06.137931 22514 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.236500 21339 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:06.275200 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:06.276052 21413 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:06.250322 22691 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:06.245635 21408 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:06.190655 22681 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:06.247315 22827 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.229314 22760 raft_consensus.cc:697] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:06.251619 21164 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36406 (request call id 226) took 5727 ms. Trace:
W20251215 08:59:06.283344 21944 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.064s sys 0.092s
I20251215 08:59:06.267767 21161 catalog_manager.cc:5654] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: config changed from index 3 to 4, VOTER 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) evicted. New cstate: current_term: 1 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:06.270918 22605 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:06.289901 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:06.295487 21161 catalog_manager.cc:5654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251215 08:59:06.284171 21944 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.514s user 0.064s sys 0.092s
I20251215 08:59:06.297673 21161 catalog_manager.cc:5654] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:06.303728 21161 catalog_manager.cc:5654] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:06.283663 22733 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831) with higher term. Message: Invalid argument: T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4.
I20251215 08:59:06.312553 22760 consensus_queue.cc:237] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:06.305660 21161 catalog_manager.cc:5654] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:06.300469 22605 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.273413 22604 consensus_queue.cc:237] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.327469 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1214) took 1068 ms. Trace:
W20251215 08:59:06.297734 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 532472us
I20251215 08:59:06.284121 22691 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:06.328681 21265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.023s sys 0.053s
W20251215 08:59:06.334652 21265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.023s sys 0.053s
I20251215 08:59:06.283882 21164 rpcz_store.cc:276] 1215 08:59:00.507178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:00.507288 (+ 110us) service_pool.cc:225] Handling call
1215 08:59:00.671976 (+164688us) spinlock_profiling.cc:243] Waited 48.7 ms on lock 0x7b5400030cd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398d93b13c 00007f398da3544e 00007f398bf334a4 00007f398bf33432 00007f398bf33361 00007f398bf332dc 00007f398bf32592 00007f3985ef355c 00007f3985ef2996
1215 08:59:06.234878 (+5562902us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:03.189052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:03.322403 (+133351us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:03.344330 (+ 21927us) write_op.cc:435] Acquired schema lock
1215 08:59:03.346269 (+ 1939us) tablet.cc:660] Decoding operations
1215 08:59:03.355047 (+ 8778us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:03.355135 (+ 88us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:03.355170 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
1215 08:59:03.355582 (+ 412us) tablet.cc:699] Row locks acquired
1215 08:59:03.355600 (+ 18us) write_op.cc:260] PREPARE: finished
1215 08:59:03.355759 (+ 159us) write_op.cc:270] Start()
1215 08:59:03.355936 (+ 177us) write_op.cc:276] Timestamp: P: 1765789143355694 usec, L: 0
1215 08:59:03.355972 (+ 36us) op_driver.cc:348] REPLICATION: starting
1215 08:59:03.469082 (+113110us) log.cc:844] Serialized 982 byte log entry
1215 08:59:03.628955 (+159873us) op_driver.cc:464] REPLICATION: finished
1215 08:59:03.802988 (+174033us) write_op.cc:301] APPLY: starting
1215 08:59:03.810503 (+ 7515us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:03.879263 (+ 68760us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:03.944243 (+ 64980us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:04.044066 (+ 99823us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:04.044377 (+ 311us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1215 08:59:04.044468 (+ 91us) write_op.cc:312] APPLY: finished
1215 08:59:04.044791 (+ 323us) log.cc:844] Serialized 29 byte log entry
1215 08:59:04.045056 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:04.072991 (+ 27935us) write_op.cc:454] Released schema lock
1215 08:59:04.082534 (+ 9543us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:04.106140 (+ 23606us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:04.106174 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":125782,"reported_tablets":5,"spinlock_wait_cycles":112106368,"child_traces":[["op",{"apply.queue_time_us":145749,"apply.run_cpu_time_us":3361,"apply.run_wall_time_us":318954,"num_ops":2,"prepare.queue_time_us":79782,"prepare.run_cpu_time_us":4336,"prepare.run_wall_time_us":366077,"replication_time_us":272851,"thread_start_us":78983,"threads_started":2}]]}
I20251215 08:59:06.342671 22691 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.337083 22680 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.322750 22681 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.340764 22821 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:06.324133 21161 catalog_manager.cc:5654] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:06.392763 22822 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:06.405022 21161 catalog_manager.cc:5654] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 2, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 2 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:06.407130 21161 catalog_manager.cc:5654] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:06.330387 21262 rpcz_store.cc:276] 1215 08:59:05.258987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:05.420745 (+161758us) service_pool.cc:225] Handling call
1215 08:59:05.421010 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:05.528064 (+107054us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:06.327138 (+799074us) spinlock_profiling.cc:243] Waited 95.7 ms on lock 0x7b5400324954. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:06.327385 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220042624}
I20251215 08:59:06.383780 22827 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.402400 22691 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.422511 22605 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.432397 22827 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.446597 22827 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.420706 22822 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.434725 22687 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:06.455595 22822 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:06.441908 22681 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:06.436008 22691 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.433144 22821 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.482663 21266 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:06.495186 22821 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.505378 22827 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:06.509344 22827 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.510478 22691 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:06.511076 22827 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:06.555467 21307 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.018s sys 0.021s
W20251215 08:59:06.556012 21307 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s user 0.018s sys 0.021s
W20251215 08:59:06.556697 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 680583us
I20251215 08:59:06.557627 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:06.564209 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 427145us
W20251215 08:59:06.561089 21946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.020s sys 0.022s
W20251215 08:59:06.565919 21946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.021s sys 0.022s
W20251215 08:59:06.566226 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 641662us
I20251215 08:59:06.567541 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:06.568235 22729 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:06.577205 22827 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:06.579098 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:06.577874 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 286194us
W20251215 08:59:06.583315 22751 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.007s sys 0.031s
W20251215 08:59:06.583791 22751 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.663s user 0.007s sys 0.031s
I20251215 08:59:06.590276 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:06.596731 21335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.000s sys 0.013s
W20251215 08:59:06.597223 21335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s user 0.000s sys 0.014s
W20251215 08:59:06.594468 21945 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.003s sys 0.030s
I20251215 08:59:06.595953 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:06.568629 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:06.600839 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:06.605943 21307 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:06.607257 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:06.587675 22761 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.005s sys 0.011s
I20251215 08:59:06.594208 22827 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251215 08:59:06.586617 22799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.000s sys 0.013s
I20251215 08:59:06.632243 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:06.607712 21945 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.003s sys 0.030s
I20251215 08:59:06.641110 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.641424 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.640427 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.715476 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.716509 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:06.707381 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:06.596510 22680 raft_consensus.cc:2764] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20251215 08:59:06.718132 21378 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.005s sys 0.056s
W20251215 08:59:06.634507 22831 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.001s sys 0.016s
W20251215 08:59:06.789119 21378 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.005s sys 0.056s
W20251215 08:59:06.789179 22831 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.001s sys 0.016s
I20251215 08:59:06.789570 21972 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:06.715147 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:06.781944 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.791145 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:06.636029 22761 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s user 0.006s sys 0.011s
I20251215 08:59:06.631711 22760 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.783857 21338 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:06.632007 22654 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:06.824483 22654 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:06.825047 22654 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:06.829093 22860 log.cc:927] Time spent T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.198s user 0.000s sys 0.001s
W20251215 08:59:06.781769 21341 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.015s sys 0.035s
W20251215 08:59:06.830209 21341 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.739s user 0.016s sys 0.035s
W20251215 08:59:06.635923 22799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.000s sys 0.013s
I20251215 08:59:06.632648 22751 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:06.620026 22656 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:06.789633 22831 raft_consensus.cc:697] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
W20251215 08:59:06.783464 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:06.792037 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.792413 21336 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:06.794059 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:06.851562 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:06.803447 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:06.637363 22609 consensus_queue.cc:1059] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.179s
I20251215 08:59:06.632202 22835 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:06.791724 21333 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:06.822711 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:06.831430 21379 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.022s sys 0.032s
I20251215 08:59:06.631285 22738 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.161s
I20251215 08:59:06.798202 22827 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:06.868896 21379 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.025s sys 0.036s
W20251215 08:59:06.871295 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 750230us
I20251215 08:59:06.805717 22680 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:06.834295 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:06.817873 21943 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.004s sys 0.022s
W20251215 08:59:06.845917 21263 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.007s sys 0.033s
W20251215 08:59:06.911404 21263 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.791s user 0.007s sys 0.033s
I20251215 08:59:06.889712 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.914171 21340 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:06.783149 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:06.870748 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:06.915858 21977 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251215 08:59:06.915895 21338 raft_consensus.cc:2410] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:06.926044 21980 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:06.927336 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:06.927748 22854 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20251215 08:59:06.927640 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:06.928459 22836 log.cc:927] Time spent T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20251215 08:59:06.876794 22827 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:06.844744 22751 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:06.845968 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:06.843919 22760 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:06.879516 22604 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:07.073213 22766 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.030s sys 0.046s
I20251215 08:59:07.118089 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:07.022189 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:07.314261 22853 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20251215 08:59:07.321920 22604 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.322227 21333 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:07.067242 22837 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.256s
W20251215 08:59:07.296892 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:07.323721 22660 raft_consensus.cc:3055] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:07.315327 21379 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:07.324182 22660 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:07.325316 22853 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:06.924022 22835 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:07.325203 22660 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:07.321511 22766 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.061s user 0.046s sys 0.078s
W20251215 08:59:06.911317 21943 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s user 0.004s sys 0.028s
I20251215 08:59:07.322113 21975 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:59:07.325939 22853 raft_consensus.cc:697] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:07.136767 21944 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.004s sys 0.024s
W20251215 08:59:07.390446 21944 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.955s user 0.007s sys 0.027s
W20251215 08:59:07.390826 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 1.01534e+06us
I20251215 08:59:07.391114 22398 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:07.391745 22398 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:06.929447 22827 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:07.393565 22848 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:07.394084 22848 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.394964 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:07.395658 21339 raft_consensus.cc:2368] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3.
I20251215 08:59:07.396209 22848 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:07.156210 22829 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.164s user 0.036s sys 0.017s
I20251215 08:59:07.314250 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:06.915500 22680 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:07.144237 21381 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.034s user 0.034s sys 0.024s
W20251215 08:59:07.406289 21381 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.296s user 0.035s sys 0.028s
I20251215 08:59:06.969601 22751 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:07.107667 22536 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.003s sys 0.069s
I20251215 08:59:07.409852 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:07.396806 22848 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:07.343328 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:07.346515 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:07.347779 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:07.411659 21338 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
W20251215 08:59:06.929437 22855 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
W20251215 08:59:07.023025 22681 consensus_meta.cc:287] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.563s user 0.014s sys 0.000s
I20251215 08:59:07.382725 21333 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:06.873874 22839 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:07.392967 22761 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:06.917608 21971 raft_consensus.cc:1275] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251215 08:59:07.256436 22606 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.092s user 0.021s sys 0.031s
I20251215 08:59:07.396708 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:07.257297 21336 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:07.422626 21265 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251215 08:59:07.320845 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 308797us
I20251215 08:59:07.326164 22835 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:07.257999 21976 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:07.338110 22604 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:07.407995 21341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.003s sys 0.008s
I20251215 08:59:07.324952 22760 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.335705 22660 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:07.412556 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:07.433303 21338 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:07.059649 22609 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:07.416061 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:07.404536 22829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.412s user 0.038s sys 0.023s
I20251215 08:59:07.397284 22837 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:07.398310 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:07.409716 22536 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.006s user 0.008s sys 0.071s
W20251215 08:59:07.424438 21263 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.017s sys 0.005s
I20251215 08:59:07.425796 21339 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:07.440248 21263 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.525s user 0.017s sys 0.005s
W20251215 08:59:07.425541 22606 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.257s user 0.021s sys 0.034s
I20251215 08:59:07.418668 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:07.426951 22799 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:07.431092 21341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.003s sys 0.008s
I20251215 08:59:07.431566 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:07.434058 21266 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:07.434816 21304 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.007s sys 0.021s
W20251215 08:59:07.461694 21304 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.543s user 0.007s sys 0.021s
W20251215 08:59:07.462097 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 557247us
I20251215 08:59:07.438944 22760 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:07.439148 22837 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.426333 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:07.419668 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:07.441262 21263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1284) took 1391 ms. Trace:
I20251215 08:59:07.441715 21261 raft_consensus.cc:1275] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:07.446038 21980 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:07.470144 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1290) took 1380 ms. Trace:
I20251215 08:59:07.470683 21341 rpcz_store.cc:276] 1215 08:59:06.089857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:06.089963 (+ 106us) service_pool.cc:225] Handling call
1215 08:59:06.090206 (+ 243us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:06.090424 (+ 218us) raft_consensus.cc:1534] Early marking committed up to index 2
1215 08:59:06.846612 (+756188us) log.cc:844] Serialized 14 byte log entry
1215 08:59:07.469822 (+623210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:07.469873 (+ 51us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:07.469913 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:07.469976 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:07.470080 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":976646,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":1353040,"threads_started":2}
I20251215 08:59:07.068450 22831 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.473651 22729 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:07.473799 22857 log.cc:927] Time spent T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.300s user 0.001s sys 0.000s
I20251215 08:59:07.474232 22729 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.430583 21378 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:07.469483 21263 rpcz_store.cc:276] 1215 08:59:06.049764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:06.049931 (+ 167us) service_pool.cc:225] Handling call
1215 08:59:06.062350 (+ 12419us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:06.072664 (+ 10314us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:06.072698 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:06.076698 (+ 4000us) log.cc:844] Serialized 26 byte log entry
1215 08:59:06.911895 (+835197us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:07.440731 (+528836us) log.cc:844] Serialized 14 byte log entry
1215 08:59:07.440923 (+ 192us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:07.441064 (+ 141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:07.441104 (+ 40us) raft_consensus.cc:1685] finished
1215 08:59:07.441116 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:07.441214 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4539,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":3415,"thread_start_us":1317212,"threads_started":2,"wal-append.queue_time_us":1157634}
W20251215 08:59:07.477826 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 47373.7us
I20251215 08:59:07.420647 22848 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.440013 22604 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:07.428371 22839 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.457404 21972 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:59:07.499065 22848 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.446002 22609 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.447747 22761 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.501536 22609 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.502123 22761 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.502669 22674 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:07.506309 22837 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:07.508486 22654 consensus_meta.cc:287] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.683s user 0.009s sys 0.002s
I20251215 08:59:07.508906 22654 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.464072 22536 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:07.497937 22738 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:07.520004 22680 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.475265 22834 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:07.485280 22829 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:07.530925 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:07.497030 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:07.525682 21334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.040s sys 0.080s
I20251215 08:59:07.479327 22535 consensus_queue.cc:1048] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20251215 08:59:07.587210 21333 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251215 08:59:07.614460 21333 pending_rounds.cc:85] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 1
I20251215 08:59:07.496526 22604 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.521859 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:07.518537 22827 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.478682 22799 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.478106 22760 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.614737 21333 pending_rounds.cc:107] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251215 08:59:07.526940 22738 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.624341 21333 raft_consensus.cc:2889] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:07.588011 21339 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:07.522194 21973 raft_consensus.cc:2410] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20251215 08:59:07.612988 21334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s user 0.045s sys 0.103s
I20251215 08:59:07.526142 22536 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.490265 22853 consensus_queue.cc:237] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.634581 21340 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 3. Ops: []
I20251215 08:59:07.530234 22834 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.622419 22654 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.630493 22846 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:07.627561 21976 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:07.657847 22656 raft_consensus.cc:3055] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:07.660882 22656 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:07.617048 22829 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.662998 21976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46170 (request call id 285) took 1050 ms. Trace:
I20251215 08:59:07.663712 22604 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.694936 22834 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:07.698683 22536 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:07.701086 22829 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.659005 22846 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.719022 22738 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.719798 22839 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.718622 22729 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.693410 22656 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:07.724152 22827 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.727057 22799 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:07.731865 22859 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20251215 08:59:07.727749 22760 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.728678 22751 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.868575 22656 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:07.917156 22751 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.917382 22835 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:07.969182 21263 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:07.972223 21263 raft_consensus.cc:1275] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.015306 22398 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:08.021528 21161 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.204s user 0.025s sys 0.009s
W20251215 08:59:08.021914 21161 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.217s user 0.026s sys 0.009s
W20251215 08:59:08.069046 21306 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.275s user 0.021s sys 0.044s
W20251215 08:59:08.069417 21306 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.288s user 0.021s sys 0.044s
W20251215 08:59:08.069968 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:08.070815 22656 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.071344 22656 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.072467 22846 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.073280 22656 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.073861 22656 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.074330 22656 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.077343 22656 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.686322 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.094406 21973 raft_consensus.cc:2410] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251215 08:59:08.111533 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:08.112362 21977 raft_consensus.cc:2393] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:08.114563 22730 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:08.115100 22730 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.116454 22558 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:08.116981 22730 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.117048 22558 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:08.118348 21259 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.008s sys 0.021s
W20251215 08:59:08.118620 21259 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.652s user 0.008s sys 0.021s
I20251215 08:59:08.119814 22558 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.123034 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.123911 21974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 390) took 1037 ms. Trace:
W20251215 08:59:08.124622 21336 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.021s sys 0.044s
I20251215 08:59:08.124429 21974 rpcz_store.cc:276] 1215 08:59:07.086151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.119085 (+ 32934us) service_pool.cc:225] Handling call
1215 08:59:08.122842 (+1003757us) spinlock_profiling.cc:243] Waited 9.75 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:08.123854 (+ 1012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22435712}
W20251215 08:59:08.125187 21336 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.023s sys 0.044s
I20251215 08:59:08.127336 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1448) took 1481 ms. Trace:
I20251215 08:59:08.127739 21336 rpcz_store.cc:276] 1215 08:59:06.645969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.427220 (+781251us) service_pool.cc:225] Handling call
1215 08:59:07.427494 (+ 274us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:07.427807 (+ 313us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:07.427832 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:07.428234 (+ 402us) log.cc:844] Serialized 26 byte log entry
1215 08:59:08.125778 (+697544us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:08.126114 (+ 336us) log.cc:844] Serialized 14 byte log entry
1215 08:59:08.127044 (+ 930us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.127128 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:08.127165 (+ 37us) raft_consensus.cc:1685] finished
1215 08:59:08.127176 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:08.127282 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":201,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":2581,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":5842,"thread_start_us":659143,"threads_started":1,"wal-append.queue_time_us":460875}
W20251215 08:59:08.141396 21338 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.002s sys 0.012s
W20251215 08:59:08.141768 21338 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.523s user 0.003s sys 0.012s
I20251215 08:59:08.142989 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.143702 21339 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:08.144642 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.145205 21339 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:08.145906 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.146876 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.147794 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.149369 21413 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:08.151201 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.151868 21413 raft_consensus.cc:2393] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:08.151793 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:08.152493 21332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.013s sys 0.013s
W20251215 08:59:08.152793 21332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.013s sys 0.013s
I20251215 08:59:08.156168 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.156942 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.157670 21336 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:08.163216 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.163656 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.163820 21332 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:08.164721 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.165789 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.166011 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.166877 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.160527 22835 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.161679 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.168331 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:08.168637 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.157840 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.169392 21339 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:59:08.167287 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.171597 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.172521 21332 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:08.174028 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.174629 21336 raft_consensus.cc:1240] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
W20251215 08:59:08.175670 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:08.176149 21339 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.177723 21339 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:08.179045 22609 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:08.179220 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.179584 22609 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.181593 22609 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.181644 21266 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.182196 22609 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.182711 22609 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.184610 22609 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.185405 22656 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.185986 22656 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.188253 22656 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:08.185427 21307 connection.cc:232] RPC call timeout handler was delayed by -1.7965160470000683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
I20251215 08:59:08.190305 22656 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.190760 22656 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.192462 22656 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.193264 22876 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.193797 22656 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.193863 22876 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.194425 22656 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.196084 22876 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.196336 22656 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.196607 22876 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:08.193310 21307 connection.cc:232] RPC call timeout handler was delayed by -1.1067532210000763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
I20251215 08:59:08.197805 22876 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:08.198238 21307 connection.cc:232] RPC call timeout handler was delayed by -1.0414160470000979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2643s for a response
I20251215 08:59:08.198840 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.199532 22609 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:08.199769 22876 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:08.202389 21381 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.012s sys 0.015s
I20251215 08:59:08.202353 22876 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:08.202555 21307 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:08.202862 21381 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s user 0.012s sys 0.015s
W20251215 08:59:08.203205 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 751417us
I20251215 08:59:08.206022 21263 raft_consensus.cc:1275] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.205899 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.207140 22609 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:08.207244 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:08.194015 21232 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.016s sys 0.018s
W20251215 08:59:08.209309 21232 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s user 0.017s sys 0.018s
I20251215 08:59:08.209689 22398 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:08.210081 22398 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.210489 21339 raft_consensus.cc:1275] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.208379 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.217530 22869 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.217984 21257 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:08.218142 22869 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.219034 22609 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.220647 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.224161 21339 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.233974 22398 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.235773 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.242584 22869 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.242846 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.244633 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:08.247102 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:08.247908 21413 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:08.243273 22869 consensus_queue.cc:1059] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:08.249462 22733 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:08.236915 22872 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:08.253152 22733 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:08.254132 22872 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.254719 22733 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:08.255625 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:08.256419 21415 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:08.256369 22872 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.257629 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.258711 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.261931 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.262689 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.256785 22837 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.007s
I20251215 08:59:08.242590 22578 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20251215 08:59:08.265236 22875 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251215 08:59:08.266556 21979 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:08.268107 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.272513 22733 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.270036 22853 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20251215 08:59:08.252472 22558 consensus_queue.cc:1048] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251215 08:59:08.274725 22733 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.278261 21381 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:08.278373 22875 consensus_queue.cc:1048] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:59:08.278163 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.280961 22837 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:08.281430 22837 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:08.281773 22837 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:08.282366 21408 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:08.285140 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.294581 21413 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:08.296614 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:59:08.300573 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 88244.9us
I20251215 08:59:08.303737 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:08.306641 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:08.307361 21263 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:08.319286 21263 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
W20251215 08:59:08.330631 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 63945.8us
I20251215 08:59:08.332473 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:08.332638 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:08.332948 22895 raft_consensus.cc:2764] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251215 08:59:08.333334 22875 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:08.337987 22896 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.338042 22875 raft_consensus.cc:697] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:08.338874 22853 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:08.338675 22896 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.339254 22875 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.340602 21378 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:08.345788 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.346376 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.346755 21977 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:08.347643 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:08.347723 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:08.347930 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.349191 21973 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:08.360458 22799 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.361984 22799 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:08.362532 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:08.364070 22799 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:08.364938 22904 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:08.365475 22799 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:08.365705 22904 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.368618 22904 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.372025 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.372892 21336 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:59:08.373939 22835 consensus_queue.cc:1059] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:08.375275 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.370963 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:08.378204 21980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 388) took 1500 ms (client timeout 1533 ms). Trace:
W20251215 08:59:08.378573 21980 rpcz_store.cc:269] 1215 08:59:06.877431 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:06.926988 (+ 49557us) service_pool.cc:225] Handling call
1215 08:59:08.378033 (+1451045us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54004218d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:08.378126 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1142833920}
W20251215 08:59:08.381557 22899 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
W20251215 08:59:08.382427 21337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.005s sys 0.009s
I20251215 08:59:08.381716 22837 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:08.382778 21337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s user 0.006s sys 0.009s
I20251215 08:59:08.340787 22896 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.391111 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.394171 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.395303 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.396469 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.399677 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.403177 21263 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:08.404242 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:08.404948 21263 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:08.431867 21978 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.116s user 0.002s sys 0.027s
W20251215 08:59:08.432313 21978 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.117s user 0.002s sys 0.027s
I20251215 08:59:08.435734 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1289) took 2346 ms. Trace:
I20251215 08:59:08.437670 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:08.436254 21334 rpcz_store.cc:276] 1215 08:59:06.089275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:06.089401 (+ 126us) service_pool.cc:225] Handling call
1215 08:59:06.089704 (+ 303us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:06.089902 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 4
1215 08:59:06.595532 (+505630us) log.cc:844] Serialized 14 byte log entry
1215 08:59:07.647351 (+1051819us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:07.649897 (+ 2546us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:07.652645 (+ 2748us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.435467 (+782822us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:08.435637 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8395,"prepare.queue_time_us":976954,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":13064,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":5221,"thread_start_us":1520337,"threads_started":2,"wal-append.queue_time_us":1262381}
I20251215 08:59:08.433548 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 389) took 1492 ms. Trace:
I20251215 08:59:08.433539 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.460387 21263 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:08.455796 21978 rpcz_store.cc:276] 1215 08:59:06.940941 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:06.944511 (+ 3570us) service_pool.cc:225] Handling call
1215 08:59:06.968482 (+ 23971us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:07.314435 (+345953us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:07.314480 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:07.315070 (+ 590us) log.cc:844] Serialized 26 byte log entry
1215 08:59:08.432672 (+1117602us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:08.433021 (+ 349us) log.cc:844] Serialized 14 byte log entry
1215 08:59:08.433236 (+ 215us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.433304 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:08.433339 (+ 35us) raft_consensus.cc:1685] finished
1215 08:59:08.433352 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:08.433470 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":159,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":351995,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"thread_start_us":1117158,"threads_started":1,"wal-append.queue_time_us":555735}
I20251215 08:59:08.475342 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.479081 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:08.482895 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:08.484791 21263 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:08.485128 21265 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:08.492044 21265 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:08.498315 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.499570 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:08.500193 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 107799us
I20251215 08:59:08.500747 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.501734 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.502836 21307 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.507483 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:08.505921 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 135550us
I20251215 08:59:08.503647 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.513096 21378 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:08.572463 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:08.603767 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 86209.4us
W20251215 08:59:08.505285 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 89815.7us
I20251215 08:59:08.505473 21263 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:08.604848 21944 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:08.506156 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.607897 22916 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:08.608281 22919 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:08.609623 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:08.610713 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 103349us
I20251215 08:59:08.611372 21307 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:08.612282 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.612462 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.614370 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:08.616168 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.616986 21263 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:08.610212 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1298) took 1092 ms. Trace:
I20251215 08:59:08.626636 21337 rpcz_store.cc:276] 1215 08:59:07.518141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.519054 (+ 913us) service_pool.cc:225] Handling call
1215 08:59:07.519482 (+ 428us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:07.519824 (+ 342us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:07.519852 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:07.520316 (+ 464us) log.cc:844] Serialized 26 byte log entry
1215 08:59:08.384584 (+864268us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:08.384870 (+ 286us) log.cc:844] Serialized 14 byte log entry
1215 08:59:08.609973 (+225103us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.610026 (+ 53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:08.610056 (+ 30us) raft_consensus.cc:1685] finished
1215 08:59:08.610063 (+ 7us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:08.610154 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4587,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":1055,"raft.queue_time_us":115628,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":7264,"thread_start_us":1022743,"threads_started":2,"wal-append.queue_time_us":389362}
W20251215 08:59:08.631644 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 166483us
I20251215 08:59:08.638756 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.640429 21265 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:08.641628 21307 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:08.643786 21307 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.620703 22919 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.632844 22916 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:08.654596 22916 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:08.655232 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.645952 22912 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
W20251215 08:59:08.669623 21379 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.286s user 0.022s sys 0.037s
W20251215 08:59:08.670120 21379 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.287s user 0.022s sys 0.037s
I20251215 08:59:08.670506 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.660756 22919 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:08.647784 22870 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:08.670506 22912 raft_consensus.cc:697] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:08.672104 21229 connection.cc:232] RPC call timeout handler was delayed by -1.1052568019999853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20251215 08:59:08.672227 21260 raft_consensus.cc:2955] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:08.660899 22916 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:08.689316 22912 consensus_queue.cc:237] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.685827 22870 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:08.720643 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 76936.8us
I20251215 08:59:08.723574 22927 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:08.725253 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.724998 22927 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.726509 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.727252 22876 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:08.727633 22927 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.727908 22876 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:08.728459 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.730142 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.730834 22926 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:08.730834 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.731788 22876 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.724052 22921 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:08.732380 22921 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.731369 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.747802 22876 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:08.728938 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1289) took 1306 ms. Trace:
I20251215 08:59:08.753186 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.748320 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.753947 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.755100 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:08.732777 22921 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:08.758128 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.758458 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.759680 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.760375 21413 raft_consensus.cc:2393] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:08.762485 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.763072 21407 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:08.763790 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.764566 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:08.766506 21341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.152s user 0.005s sys 0.012s
W20251215 08:59:08.766876 21341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.153s user 0.005s sys 0.012s
I20251215 08:59:08.767578 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1295) took 1273 ms. Trace:
I20251215 08:59:08.767834 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:08.768517 21338 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:08.768692 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1297) took 1331 ms. Trace:
I20251215 08:59:08.772464 21264 rpcz_store.cc:276] 1215 08:59:07.437742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.438803 (+ 1061us) service_pool.cc:225] Handling call
1215 08:59:07.439089 (+ 286us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:08.138869 (+699780us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:08.138904 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:08.138953 (+ 49us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:59:08.138987 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.768483 (+629496us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b54002e59d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:08.768528 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:08.768642 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1608740736}
I20251215 08:59:08.774605 21412 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.769167 22919 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.767881 21341 rpcz_store.cc:276] 1215 08:59:07.494574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.520866 (+ 26292us) service_pool.cc:225] Handling call
1215 08:59:07.521319 (+ 453us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:07.521696 (+ 377us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:07.521735 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:07.527414 (+ 5679us) log.cc:844] Serialized 26 byte log entry
1215 08:59:08.767274 (+1239860us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:08.767314 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.767376 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:08.767418 (+ 42us) raft_consensus.cc:1685] finished
1215 08:59:08.767429 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:08.767530 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5099,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":54972,"thread_start_us":1152916,"threads_started":1,"wal-append.queue_time_us":331564}
I20251215 08:59:08.771067 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.776257 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.777590 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.778110 21978 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20251215 08:59:08.778548 21978 pending_rounds.cc:85] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Aborting all ops after (but not including) 4
I20251215 08:59:08.778221 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.778754 21978 pending_rounds.cc:107] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251215 08:59:08.779142 21978 raft_consensus.cc:2889] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:08.779048 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.779914 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.782939 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.765758 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:08.786072 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.786702 21413 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:08.787998 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.788666 21413 raft_consensus.cc:2393] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:08.764915 22876 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.791421 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.777856 22926 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:08.802542 21339 raft_consensus.cc:1275] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251215 08:59:08.803615 21232 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:08.732923 22870 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.790069 22919 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:08.813957 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:08.749655 21262 rpcz_store.cc:276] 1215 08:59:07.422485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.425260 (+ 2775us) service_pool.cc:225] Handling call
1215 08:59:07.429666 (+ 4406us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:08.726662 (+1296996us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:08.726718 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:08.726782 (+ 64us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:08.726839 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.728633 (+ 1794us) spinlock_profiling.cc:243] Waited 83.4 ms on lock 0x7b5400323554. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:08.728685 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:08.728863 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191818112}
I20251215 08:59:08.813982 22902 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20251215 08:59:08.806851 22904 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:08.806463 22876 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:08.818149 22918 log.cc:927] Time spent T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20251215 08:59:08.818466 22876 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.813213 22919 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.820459 22876 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.820549 21408 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:08.820648 22919 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:08.823109 22925 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:08.796248 22926 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:08.814558 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.838124 21332 raft_consensus.cc:1275] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.818123 22904 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:08.841554 22904 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:08.841699 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.842686 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:08.836844 22903 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:08.843312 21978 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:08.843493 22903 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:08.844081 22903 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:08.844937 21378 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:08.841583 22870 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.849222 22930 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:08.849725 22930 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.851737 22930 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:08.851584 22870 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:08.861994 22936 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:08.862552 22936 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:08.862943 22936 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:08.835198 22926 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:08.872750 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 66372.8us
I20251215 08:59:08.875638 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.875847 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.878736 21381 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:08.823604 22925 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.884080 21266 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:08.880882 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:08.883450 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 69554.6us
I20251215 08:59:08.883693 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:08.871385 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:08.880286 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:08.891335 21406 raft_consensus.cc:606] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:08.891736 21406 raft_consensus.cc:614] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:08.894371 21259 raft_consensus.cc:1275] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.883020 22876 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.899482 22935 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.899294 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2c10c3de775846868774c98cc0a75094"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:08.903780 22904 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.914839 22904 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:07.688019 21976 rpcz_store.cc:276] 1215 08:59:06.611308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:06.634524 (+ 23216us) service_pool.cc:225] Handling call
1215 08:59:07.661836 (+1027312us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:08.905788 22938 consensus_queue.cc:1059] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:08.916054 21264 raft_consensus.cc:2393] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:08.915951 22938 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:08.916488 21406 raft_consensus.cc:606] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:08.918681 22938 consensus_queue.cc:1048] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:08.895156 22940 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:08.921191 22903 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.922768 21971 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:08.931818 22945 raft_consensus.cc:3055] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
W20251215 08:59:08.932405 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54410us
I20251215 08:59:08.881780 22916 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.941324 22925 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.935205 22935 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:08.960016 22935 raft_consensus.cc:697] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:08.961251 22935 consensus_queue.cc:237] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.935030 22876 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:08.979867 22921 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.981307 21980 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:08.936479 22945 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:08.981685 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 102980us
I20251215 08:59:08.983410 22903 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:08.985105 21261 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:08.957264 22912 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:08.985716 21261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1299) took 1017 ms. Trace:
I20251215 08:59:08.987174 21261 rpcz_store.cc:276] 1215 08:59:07.967886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:08.984441 (+1016555us) service_pool.cc:225] Handling call
1215 08:59:08.984752 (+ 311us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:08.985481 (+ 729us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.985663 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:08.975467 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:08.965526 22876 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.995652 22876 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.988505 22912 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.996739 22926 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:08.996968 22398 raft_consensus.cc:3055] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:09.001416 22398 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:09.002499 22398 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.003466 22398 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.001335 22926 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.993573 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.009936 22945 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:09.011794 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:09.012905 22945 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.000591 22936 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:08.998133 22916 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:08.998958 22606 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251215 08:59:08.999912 22919 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.027422 22919 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.000730 22742 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.012739 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:08.998039 22921 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.033066 22936 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.000274 22912 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:09.034667 22835 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.026s sys 0.033s
W20251215 08:59:09.035152 22835 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.026s sys 0.033s
I20251215 08:59:09.035071 22912 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.035805 22912 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:09.043071 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 148072us
I20251215 08:59:09.045197 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:09.048777 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.052608 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.053282 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.054198 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.053344 21414 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:09.055449 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.056002 21264 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:09.057508 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:09.058151 21264 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.055840 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:09.068080 21261 raft_consensus.cc:2368] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3.
I20251215 08:59:09.071856 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:09.072388 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 91285.4us
I20251215 08:59:09.072775 21259 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.074564 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.078411 21264 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:09.053344 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.080804 21408 raft_consensus.cc:2393] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 2: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:59:09.081662 21230 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.691s user 0.007s sys 0.018s
W20251215 08:59:09.081986 21230 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.691s user 0.007s sys 0.018s
W20251215 08:59:09.082473 21230 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.899s (SENT) [suppressed 31 similar messages]
W20251215 08:59:09.083050 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:09.084072 21230 leader_election.cc:336] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.803s (SENT)
I20251215 08:59:09.066040 22916 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:09.072755 22926 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:09.085460 22926 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.085353 22916 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.086840 22398 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.087190 22926 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.087321 22398 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.087327 22916 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.089226 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.067047 22919 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:09.089236 22398 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:09.090912 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:09.094780 21259 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:09.099846 22950 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:09.099110 21230 connection.cc:232] RPC call timeout handler was delayed by -1.7604675300000281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20251215 08:59:09.101358 21230 connection.cc:232] RPC call timeout handler was delayed by -1.6856714709999778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
W20251215 08:59:09.109242 21230 connection.cc:232] RPC call timeout handler was delayed by -1.6128668640000114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
W20251215 08:59:09.109846 21230 connection.cc:232] RPC call timeout handler was delayed by -1.6073714709999649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2143s for a response
W20251215 08:59:09.110263 21230 connection.cc:232] RPC call timeout handler was delayed by -1.6060965010000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
I20251215 08:59:09.100386 22950 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:09.116914 21230 leader_election.cc:336] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.806s (SENT)
I20251215 08:59:09.117825 21230 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.119923 22578 consensus_queue.cc:1048] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:09.117715 22919 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:09.124205 21230 leader_election.cc:336] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.820s (SENT)
I20251215 08:59:09.124248 22919 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.124889 22950 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:09.125031 21230 connection.cc:232] RPC call timeout handler was delayed by -1.5833288920000541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20251215 08:59:09.125563 21230 leader_election.cc:336] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.026s (SENT)
I20251215 08:59:09.125726 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.125905 21230 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.126310 22919 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.126760 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.127197 22904 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:09.127411 21336 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:09.127790 22904 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:09.129660 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.130136 21975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 335) took 1737 ms. Trace:
I20251215 08:59:09.130493 21975 rpcz_store.cc:276] 1215 08:59:07.392473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.129320 (+1736847us) service_pool.cc:225] Handling call
1215 08:59:09.129647 (+ 327us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:09.129826 (+ 179us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:09.129852 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:09.129907 (+ 55us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:09.129936 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:09.129988 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:09.130089 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:09.132177 22912 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:09.132925 21230 connection.cc:232] RPC call timeout handler was delayed by -1.5592743470000414s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1584s for a response
I20251215 08:59:09.132972 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.130862 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:09.135214 21230 connection.cc:232] RPC call timeout handler was delayed by -1.5583965010000611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
I20251215 08:59:09.135732 21307 leader_election.cc:400] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3.
W20251215 08:59:09.136507 21230 connection.cc:232] RPC call timeout handler was delayed by -1.40049124199993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
I20251215 08:59:09.136795 21307 leader_election.cc:403] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20251215 08:59:09.136198 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 149829us
W20251215 08:59:09.136811 21230 connection.cc:232] RPC call timeout handler was delayed by -1.3697625539999763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
W20251215 08:59:09.137615 21230 connection.cc:232] RPC call timeout handler was delayed by -1.1671016550000104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
I20251215 08:59:09.137893 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.139600 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.143224 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:09.143894 21336 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:09.147531 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.149382 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:09.136834 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 56186.3us
I20251215 08:59:09.153605 21230 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.154974 22398 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:09.155411 22398 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.155810 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.156809 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.155838 22398 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.158145 22919 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.159327 21230 leader_election.cc:400] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3.
I20251215 08:59:09.160035 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.160195 21230 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.160605 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.161795 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.161754 21230 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.160601 21341 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:09.162679 21338 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:09.165489 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.166043 21265 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:09.162184 22950 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:09.168831 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.153796 22926 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.170945 22939 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.171099 22926 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.196878 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:09.224735 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54623.9us
I20251215 08:59:09.191540 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.172472 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.226908 21230 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.154060 22912 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:09.225405 21379 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.162279 22916 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:09.232358 21336 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:09.233660 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.163630 22904 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.238536 22398 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.160749 21974 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:09.165192 22903 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:09.242763 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.241426 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.243388 21978 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:09.209338 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.209025 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:09.195257 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.231366 21230 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.229360 22939 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:09.229789 22912 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:09.245273 22398 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.248433 21230 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.250793 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.253672 22398 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.229207 22578 consensus_queue.cc:1048] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251215 08:59:09.244249 22903 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:09.242749 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.247020 22950 raft_consensus.cc:697] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:09.264953 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:09.261540 21979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 387) took 2390 ms (client timeout 1576 ms). Trace:
I20251215 08:59:09.262001 21333 raft_consensus.cc:1275] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:09.262758 21409 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:09.268630 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1247) took 1120 ms. Trace:
I20251215 08:59:09.249418 22904 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:09.269114 21409 rpcz_store.cc:276] 1215 08:59:08.148179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.262160 (+1113981us) service_pool.cc:225] Handling call
1215 08:59:09.262522 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:09.268318 (+ 5796us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:09.268566 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:09.266394 21979 rpcz_store.cc:269] 1215 08:59:06.870780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:06.926988 (+ 56208us) service_pool.cc:225] Handling call
1215 08:59:09.261319 (+2334331us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b54004254d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:09.261446 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1981683584}
I20251215 08:59:09.271329 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.272475 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.251796 22912 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.249984 21975 raft_consensus.cc:2376] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3. Re-sending same reply.
I20251215 08:59:09.258791 22903 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.287571 22950 consensus_queue.cc:237] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.288710 22922 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:09.289232 22922 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:09.291893 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:09.292227 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:09.297204 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.286145 22799 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:09.260685 22872 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20251215 08:59:09.280362 22926 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.279882 22916 raft_consensus.cc:697] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:09.318028 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 73403.7us
I20251215 08:59:09.361428 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:09.367101 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 118741us
I20251215 08:59:09.368271 21230 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.369247 21230 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.370342 21230 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.370486 21944 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:09.371685 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.372674 22398 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:09.373054 22398 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.373380 22398 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.373782 21230 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.377193 21307 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.317157 22799 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:09.358150 21406 raft_consensus.cc:606] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:09.372048 22904 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:09.402422 22959 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:09.404268 22904 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.404421 22959 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.404600 22910 raft_consensus.cc:3055] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:09.401729 21406 raft_consensus.cc:614] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:09.405323 22910 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:09.406747 22959 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.406431 22910 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.407853 22910 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:09.408179 22001 debug-util.cc:398] Leaking SignalData structure 0x7b08002a7700 after lost signal to thread 20112
I20251215 08:59:09.326246 22921 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:09.409112 22921 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.412345 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.412953 21264 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:09.413318 22921 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.368148 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:09.422940 22960 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:09.404701 22904 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.374402 22926 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:09.424070 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:09.380832 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.380246 22846 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:09.383172 22939 raft_consensus.cc:697] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:09.389323 22799 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:09.422014 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.427065 21264 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:09.559209 21264 raft_consensus.cc:606] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Received request to transfer leadership to TS 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.559610 21264 raft_consensus.cc:614] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:09.388918 22905 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:09.434384 22964 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:09.446781 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.449733 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.569939 21980 raft_consensus.cc:2393] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:09.456482 22398 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.457798 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.476851 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:09.437175 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.571828 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.447671 22922 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:09.476928 22926 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.515550 22869 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20251215 08:59:09.516606 22966 log.cc:927] Time spent T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20251215 08:59:09.506737 22940 consensus_queue.cc:1048] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:09.508960 22760 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.570225 22959 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:09.576561 22959 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:09.576440 22760 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.576944 22959 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.513111 22963 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20251215 08:59:09.508193 22846 raft_consensus.cc:697] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:09.580560 21408 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:09.581190 21408 raft_consensus.cc:606] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:09.580890 22846 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.581262 22912 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.583022 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:59:09.583299 22912 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.505245 22950 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:09.507227 22921 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:09.510398 22962 consensus_queue.cc:1059] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:09.514120 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.584632 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 336) took 2191 ms. Trace:
I20251215 08:59:09.455204 22960 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:08.301729 22609 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:09.457326 22872 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20251215 08:59:09.458710 22902 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:09.476842 22001 debug-util.cc:398] Leaking SignalData structure 0x7b08002cfb40 after lost signal to thread 21198
W20251215 08:59:09.560391 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 138457us
W20251215 08:59:09.561445 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 187875us
W20251215 08:59:09.564134 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 183257us
I20251215 08:59:09.566376 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.594542 21230 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.567148 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.595978 22609 raft_consensus.cc:697] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:09.596601 22001 debug-util.cc:398] Leaking SignalData structure 0x7b08000f7240 after lost signal to thread 21290
I20251215 08:59:09.568025 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.596971 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.597658 21978 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
W20251215 08:59:09.597860 22001 debug-util.cc:398] Leaking SignalData structure 0x7b08003ae100 after lost signal to thread 21365
I20251215 08:59:09.572415 21980 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
W20251215 08:59:09.598279 22001 debug-util.cc:398] Leaking SignalData structure 0x7b08003ae020 after lost signal to thread 21366
I20251215 08:59:09.598585 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.598739 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.599721 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.600694 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.601562 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.601650 21230 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.601997 22926 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.602394 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:09.602874 21980 raft_consensus.cc:2376] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3. Re-sending same reply.
I20251215 08:59:09.602803 22799 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.573508 22398 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.604699 21408 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:09.604777 22799 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.584910 21979 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:09.511688 22939 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.588364 22960 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.606357 21972 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20251215 08:59:09.606040 22904 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.606775 21972 pending_rounds.cc:85] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Aborting all ops after (but not including) 4
I20251215 08:59:09.607069 21972 pending_rounds.cc:107] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20251215 08:59:09.607331 21972 raft_consensus.cc:2889] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:09.607987 21972 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 334) took 2215 ms. Trace:
I20251215 08:59:09.608177 22904 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:09.477296 22916 consensus_queue.cc:237] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.585129 22962 raft_consensus.cc:3055] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:09.610087 22962 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:09.596980 22609 consensus_queue.cc:237] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.610934 22962 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.612206 22962 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:09.585848 22919 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3.
I20251215 08:59:09.584640 22921 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.589776 21338 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:09.614285 22398 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:09.615945 21230 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.617794 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:09.594789 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:09.566699 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 103801us
I20251215 08:59:09.619350 21978 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
W20251215 08:59:09.622793 21335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1452) took 2813 ms (client timeout 1965 ms). Trace:
I20251215 08:59:09.594493 22959 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.595441 21304 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.597725 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:09.569171 22905 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.571000 21341 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:09.624361 21977 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:09.624917 21304 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.620530 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.625859 21304 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.626277 22398 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.626926 21304 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:09.626734 21230 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.627959 21304 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:09.623157 21335 rpcz_store.cc:269] 1215 08:59:06.800852 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.613915 (+2813063us) service_pool.cc:212] Skipping call since client already timed out
1215 08:59:09.614011 (+ 96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251215 08:59:09.630683 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.637602 21333 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:09.638025 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:09.638631 21261 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.641287 21264 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:09.638542 21229 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.636920 21304 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.642750 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.646008 22398 raft_consensus.cc:2749] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.647025 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.647025 21230 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.654575 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.648885 21977 raft_consensus.cc:1240] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20251215 08:59:09.652778 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.652392 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.667215 21413 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:09.608358 21972 rpcz_store.cc:276] 1215 08:59:07.392216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.499435 (+107219us) service_pool.cc:225] Handling call
1215 08:59:09.605805 (+2106370us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:09.605971 (+ 166us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:09.607769 (+ 1798us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:09.607930 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26328832}
W20251215 08:59:09.652252 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 58140.4us
I20251215 08:59:09.664659 21229 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.621197 22902 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:09.629518 22742 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.686841 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.687739 21229 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:09.652732 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.685910 22398 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.634907 22964 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:09.694137 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.694402 21229 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:09.691922 21261 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:09.630406 22959 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.638316 22950 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.629137 22926 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:09.629626 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.688697 21412 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.634693 22876 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:09.695590 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.708662 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.709676 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.696228 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.639523 22905 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:09.618829 21974 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:09.698215 22398 raft_consensus.cc:2764] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20251215 08:59:09.630095 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.698112 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:09.713925 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.699931 21229 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:09.699856 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 52899.2us
I20251215 08:59:09.716001 21261 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:09.716197 21230 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.622712 21334 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:09.639879 22916 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:09.645335 22904 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.647954 22896 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:09.700639 22976 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:09.648156 22940 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:09.648334 22963 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:09.684157 22920 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:09.686805 22902 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.715250 21407 raft_consensus.cc:2393] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:09.699425 22912 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:09.730202 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.718691 21229 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.718936 21230 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.587527 21971 rpcz_store.cc:276] 1215 08:59:07.392712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.655883 (+263171us) service_pool.cc:225] Handling call
1215 08:59:07.656342 (+ 459us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:08.937059 (+1280717us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:09.572793 (+635734us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54005fdf54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:09.584574 (+ 11781us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2911326464}
I20251215 08:59:09.719512 22398 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.719867 22950 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:09.649248 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.704164 22926 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:09.721201 22939 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:09.722257 22959 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20251215 08:59:09.732164 22922 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20251215 08:59:09.732542 22983 log.cc:927] Time spent T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20251215 08:59:09.732440 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:09.769640 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:09.770255 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.733582 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.736701 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.720404 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.729823 22902 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.719905 22916 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:09.726826 22976 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:09.739151 22837 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.749087 21266 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:09.757555 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:09.755465 21415 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:09.731500 21408 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:59:09.764179 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:09.764318 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:09.765581 22926 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:09.739133 22733 consensus_queue.cc:1048] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20251215 08:59:09.764053 22950 raft_consensus.cc:697] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:09.746798 21265 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:09.757462 22896 raft_consensus.cc:2764] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251215 08:59:09.757743 22872 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:09.765156 21410 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.791563 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:09.792622 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.793318 21414 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:09.777760 22916 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.784358 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:09.745592 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.799834 21261 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:09.721454 22578 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20251215 08:59:09.795107 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:09.802814 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:09.754046 22963 raft_consensus.cc:697] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:09.755916 22935 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20251215 08:59:09.779574 22976 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:09.781726 22606 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:09.805480 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:09.811653 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:09.801425 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.786234 22876 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.739825 22920 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.814846 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.815982 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.813211 22609 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:09.823295 22760 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:09.823765 22982 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20251215 08:59:09.826292 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1430) took 3184 ms. Trace:
I20251215 08:59:09.826701 21340 rpcz_store.cc:276] 1215 08:59:06.641583 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:07.419777 (+778194us) service_pool.cc:225] Handling call
1215 08:59:07.420157 (+ 380us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:07.637305 (+217148us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:09.826009 (+2188704us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b5400163854. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:09.826226 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":492552576}
I20251215 08:59:09.756466 22960 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.807688 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.831426 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:09.832211 21261 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:09.832351 21259 raft_consensus.cc:2368] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:09.793530 22872 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:09.842527 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.845995 21261 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:09.828670 22606 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.848120 21414 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:09.848867 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:09.849470 21265 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:09.855654 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:09.844137 22674 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:09.834533 22739 consensus_queue.cc:1048] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20251215 08:59:09.873689 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:09.874774 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:09.845309 22872 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:09.914670 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:09.785038 22973 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20251215 08:59:09.850914 22905 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:09.918780 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:09.924670 22928 log.cc:927] Time spent T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20251215 08:59:09.950284 21265 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
W20251215 08:59:09.950562 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 303486us
I20251215 08:59:09.960057 21265 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:09.962082 21304 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:09.940210 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:09.968286 22960 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:09.945773 22742 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:09.970374 21975 raft_consensus.cc:3055] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:09.950416 22835 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:09.978703 21163 catalog_manager.cc:5654] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:09.987258 21412 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:09.918566 22950 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:09.989287 21407 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:09.995215 22001 debug-util.cc:398] Leaking SignalData structure 0x7b0800392d20 after lost signal to thread 21439
I20251215 08:59:09.986833 21975 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
W20251215 08:59:10.010212 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 50165.1us
I20251215 08:59:09.925685 22963 consensus_queue.cc:237] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:10.010004 22001 debug-util.cc:398] Leaking SignalData structure 0x7b0800392e00 after lost signal to thread 21440
I20251215 08:59:10.034155 21381 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:10.007035 22976 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.033067 22920 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:10.062666 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 197851us
I20251215 08:59:10.067411 21263 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20251215 08:59:10.069546 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 154962us
I20251215 08:59:10.070632 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:10.070677 21946 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.053169 22935 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.079510 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:10.036756 22660 consensus_queue.cc:1059] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20251215 08:59:10.057240 22846 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.087220 21946 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.088905 21946 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.040051 22997 consensus_queue.cc:1059] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20251215 08:59:10.083993 22869 raft_consensus.cc:2764] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20251215 08:59:10.090539 22988 log.cc:927] Time spent T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.169s user 0.001s sys 0.000s
W20251215 08:59:10.107540 22001 debug-util.cc:398] Leaking SignalData structure 0x7b08003398e0 after lost signal to thread 22004
I20251215 08:59:10.076036 22922 consensus_queue.cc:1059] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20251215 08:59:10.085186 22905 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.088256 22939 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
W20251215 08:59:10.101250 22986 log.cc:927] Time spent T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20251215 08:59:10.140971 21337 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:10.141631 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1319) took 1323 ms. Trace:
I20251215 08:59:10.141891 21337 rpcz_store.cc:276] 1215 08:59:08.817775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:08.821516 (+ 3741us) service_pool.cc:225] Handling call
1215 08:59:10.140634 (+1319118us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b5800014198. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:10.140819 (+ 185us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:10.141416 (+ 597us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.141582 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9970560}
W20251215 08:59:10.143963 23005 log.cc:927] Time spent T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20251215 08:59:10.144125 22976 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:10.145923 21973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 349) took 1778 ms. Trace:
I20251215 08:59:10.146334 21973 rpcz_store.cc:276] 1215 08:59:08.367741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:08.367898 (+ 157us) service_pool.cc:225] Handling call
1215 08:59:08.368375 (+ 477us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:08.382978 (+ 14603us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:08.383015 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:08.402870 (+ 19855us) log.cc:844] Serialized 26 byte log entry
1215 08:59:09.851204 (+1448334us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b1800393280. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:10.040535 (+189331us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:10.043760 (+ 3225us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.056683 (+ 12923us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:10.145618 (+ 88935us) raft_consensus.cc:1685] finished
1215 08:59:10.145642 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:10.145857 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1173,"raft.queue_time_us":16326,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8766,"spinlock_wait_cycles":1961921280,"thread_start_us":170392,"threads_started":1,"wal-append.queue_time_us":1661601}
I20251215 08:59:10.147392 21410 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
W20251215 08:59:10.129359 22001 debug-util.cc:398] Leaking SignalData structure 0x7b08003602e0 after lost signal to thread 22005
I20251215 08:59:10.107578 23008 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.153468 22962 raft_consensus.cc:3055] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
W20251215 08:59:10.161159 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 94187.4us
I20251215 08:59:10.096668 22846 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.164423 21415 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:10.139218 22896 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.145929 23004 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:10.074735 21975 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.150671 22939 raft_consensus.cc:697] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:10.157827 22799 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:10.164690 22920 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.166085 22660 raft_consensus.cc:3055] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:10.163996 22962 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:10.191080 22660 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:10.191964 21975 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:10.194595 22927 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:10.178803 22896 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.222636 21339 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:10.284540 21232 leader_election.cc:336] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.584s (SENT)
W20251215 08:59:10.289987 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:10.294080 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1321) took 1475 ms. Trace:
I20251215 08:59:10.294613 21339 rpcz_store.cc:276] 1215 08:59:08.818451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:08.821886 (+ 3435us) service_pool.cc:225] Handling call
1215 08:59:10.222122 (+1400236us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b5800014198. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:10.222386 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:10.293288 (+ 70902us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.294006 (+ 718us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9401600}
W20251215 08:59:10.297498 22903 consensus_meta.cc:287] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.001s user 0.009s sys 0.004s
I20251215 08:59:10.207517 22916 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.207628 22660 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:10.316862 22995 log.cc:927] Time spent T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20251215 08:59:10.318151 22896 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.297906 22903 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:10.314282 22902 consensus_meta.cc:287] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.533s user 0.012s sys 0.000s
I20251215 08:59:10.322719 22902 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.325555 21262 raft_consensus.cc:1240] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:10.327473 22926 consensus_meta.cc:287] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.532s user 0.005s sys 0.004s
I20251215 08:59:10.292109 22927 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.336112 21259 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:10.338670 22916 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:10.298647 22962 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.345885 22962 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:10.355543 22927 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:10.311522 22939 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.329538 22902 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:10.361790 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 272948us
I20251215 08:59:10.328038 22926 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:10.365298 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 182689us
I20251215 08:59:10.374570 21946 leader_election.cc:400] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:10.375320 21946 leader_election.cc:403] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251215 08:59:10.379597 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:10.383666 22831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.006s user 0.055s sys 0.125s
I20251215 08:59:10.386296 22903 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:10.391757 22998 log.cc:927] Time spent T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.239s user 0.002s sys 0.000s
I20251215 08:59:10.394706 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1328) took 1138 ms. Trace:
I20251215 08:59:10.395031 21336 rpcz_store.cc:276] 1215 08:59:09.256306 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.256517 (+ 211us) service_pool.cc:225] Handling call
1215 08:59:09.256797 (+ 280us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:09.374859 (+118062us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:09.375281 (+ 422us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:09.375692 (+ 411us) log.cc:844] Serialized 26 byte log entry
1215 08:59:09.599824 (+224132us) spinlock_profiling.cc:243] Waited 7.96 ms on lock 0x7b1800338b00. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b06147 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:09.775884 (+176060us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b1800338b00. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:10.012596 (+236712us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:59:10.012654 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.012750 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:10.390046 (+377296us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:10.394549 (+ 4503us) raft_consensus.cc:1685] finished
1215 08:59:10.394570 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:10.394659 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101041,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":10435,"spinlock_wait_cycles":31940224,"thread_start_us":351554,"threads_started":2,"wal-append.queue_time_us":713601}
I20251215 08:59:10.397923 21232 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:10.393734 23004 consensus_queue.cc:1059] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20251215 08:59:10.401824 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:10.402827 23004 raft_consensus.cc:3055] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:10.402952 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.390928 22920 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:10.412647 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:10.413602 21974 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:10.416719 22733 raft_consensus.cc:993] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:10.404397 22903 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:10.403200 23004 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:10.420573 22872 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.443563 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.444018 23004 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.444940 22872 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.445302 23004 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:10.445580 21258 raft_consensus.cc:1240] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:10.420953 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:10.449220 22926 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:10.452311 21975 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:10.414577 22927 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:10.454365 22927 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.459996 21257 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:10.413928 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.405047 23008 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:10.419818 22733 raft_consensus.cc:1081] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:10.429418 22910 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.429884 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.430471 21946 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:10.430626 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:10.432405 21266 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:10.434142 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.435876 22903 raft_consensus.cc:697] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:10.437127 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1473) took 1073 ms. Trace:
I20251215 08:59:10.531080 21338 rpcz_store.cc:276] 1215 08:59:09.363931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.446538 (+ 82607us) service_pool.cc:225] Handling call
1215 08:59:10.436887 (+990349us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b5400164754. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:10.437053 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46292864}
I20251215 08:59:10.439013 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1331) took 1181 ms. Trace:
I20251215 08:59:10.533293 21333 rpcz_store.cc:276] 1215 08:59:09.257664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.569952 (+312288us) service_pool.cc:225] Handling call
1215 08:59:09.594766 (+ 24814us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:09.644388 (+ 49622us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.438723 (+794335us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b540009ced4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:10.438944 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97933696}
I20251215 08:59:10.440470 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.424336 21163 catalog_manager.cc:5654] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:10.456761 22927 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:10.493274 22831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.116s user 0.055s sys 0.126s
I20251215 08:59:10.493752 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1334) took 1199 ms. Trace:
I20251215 08:59:08.301842 22656 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.484472 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1231) took 2119 ms. Trace:
I20251215 08:59:10.544876 21341 rpcz_store.cc:276] 1215 08:59:09.294030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.624813 (+330783us) service_pool.cc:225] Handling call
1215 08:59:09.629290 (+ 4477us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:09.708340 (+ 79050us) raft_consensus.cc:1534] Early marking committed up to index 6
1215 08:59:09.719534 (+ 11194us) log.cc:844] Serialized 14 byte log entry
1215 08:59:09.775458 (+ 55924us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:09.775520 (+ 62us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:09.775571 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.493438 (+717867us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b54002111d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:10.493501 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:10.493664 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11074,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2986,"raft.queue_time_us":98,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":4014,"spinlock_wait_cycles":181342464,"thread_start_us":55550,"threads_started":1,"wal-append.queue_time_us":75757}
W20251215 08:59:10.481860 21334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1460) took 1339 ms (client timeout 1729 ms). Trace:
W20251215 08:59:10.496631 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 318081us
W20251215 08:59:10.524569 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 94174.6us
I20251215 08:59:10.480171 22846 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.545145 21260 rpcz_store.cc:276] 1215 08:59:08.365139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:08.644429 (+279290us) service_pool.cc:225] Handling call
1215 08:59:08.644727 (+ 298us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:08.647372 (+ 2645us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:08.681226 (+ 33854us) log.cc:844] Serialized 14 byte log entry
1215 08:59:08.750830 (+ 69604us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:08.751364 (+ 534us) log.cc:844] Serialized 26 byte log entry
1215 08:59:08.751496 (+ 132us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:08.751531 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:08.751602 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:10.482799 (+1731197us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x7b24000f4e68. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f398495a9a4 00007f3984958aab 00007f39849e79a9 00007f39849e7635 00007f39849e6b90 00007f398777b828 00007f398777bf88 00007f3987753738 00007f398774eb0a 00007f398774df56
1215 08:59:10.484089 (+ 1290us) raft_consensus.cc:1685] finished
1215 08:59:10.484217 (+ 128us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:10.484401 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1067,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":8802,"spinlock_wait_cycles":18359040,"thread_start_us":30571,"threads_started":1,"wal-append.queue_time_us":65826}
I20251215 08:59:10.496603 21378 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:10.554085 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.559645 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.523878 23008 raft_consensus.cc:697] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:10.563310 21946 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:10.566557 21946 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:10.567953 21946 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:10.537353 22910 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:10.559072 22846 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:10.570752 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 152946us
I20251215 08:59:10.556092 22920 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:10.570195 23015 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:10.570308 23003 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.569370 22831 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:10.570487 21976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 400) took 1403 ms (client timeout 1580 ms). Trace:
I20251215 08:59:10.575627 21411 raft_consensus.cc:2410] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:10.569595 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:10.559731 22656 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.575304 22927 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:10.575212 22799 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:10.586819 21976 rpcz_store.cc:269] 1215 08:59:09.165102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.165276 (+ 174us) service_pool.cc:225] Handling call
1215 08:59:10.430476 (+1265200us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:10.568399 (+137923us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43639040}
I20251215 08:59:10.591329 22835 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:10.593668 21411 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1437) took 1802 ms (client timeout 1584 ms). Trace:
I20251215 08:59:10.595727 23008 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.598681 22656 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.599782 22930 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:10.602257 22853 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:10.602514 21163 catalog_manager.cc:5654] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:10.603027 22674 consensus_queue.cc:1059] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:10.603569 22558 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:10.608652 22921 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:10.610924 21161 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36424 (request call id 224) took 6900 ms. Trace:
W20251215 08:59:10.613485 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:10.552913 21334 rpcz_store.cc:269] 1215 08:59:09.142792 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.160983 (+ 18191us) service_pool.cc:225] Handling call
1215 08:59:10.481619 (+1320636us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400281ed4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:10.481778 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":845661056}
I20251215 08:59:10.579733 22910 consensus_queue.cc:1059] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20251215 08:59:10.650357 21259 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:10.613828 22927 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:10.651438 21161 rpcz_store.cc:276] 1215 08:59:03.722389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:03.723004 (+ 615us) service_pool.cc:225] Handling call
1215 08:59:08.022254 (+4299250us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:08.022285 (+ 31us) write_op.cc:454] Released schema lock
1215 08:59:10.610868 (+2588583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:07.642896 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:07.671299 (+ 28403us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:07.671346 (+ 47us) write_op.cc:435] Acquired schema lock
1215 08:59:07.671370 (+ 24us) tablet.cc:660] Decoding operations
1215 08:59:07.672530 (+ 1160us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:07.672611 (+ 81us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:07.672651 (+ 40us) tablet.cc:683] Acquiring locks for 9 operations
1215 08:59:07.716461 (+ 43810us) tablet.cc:699] Row locks acquired
1215 08:59:07.717763 (+ 1302us) write_op.cc:260] PREPARE: finished
1215 08:59:07.717965 (+ 202us) write_op.cc:270] Start()
1215 08:59:07.721204 (+ 3239us) write_op.cc:276] Timestamp: P: 1765789147717915 usec, L: 0
1215 08:59:07.722119 (+ 915us) op_driver.cc:348] REPLICATION: starting
1215 08:59:07.814973 (+ 92854us) log.cc:844] Serialized 3631 byte log entry
1215 08:59:07.923716 (+108743us) op_driver.cc:464] REPLICATION: finished
1215 08:59:07.924847 (+ 1131us) write_op.cc:301] APPLY: starting
1215 08:59:07.924922 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:07.925134 (+ 212us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:07.925276 (+ 142us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:07.927044 (+ 1768us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:07.927268 (+ 224us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
1215 08:59:07.927312 (+ 44us) write_op.cc:312] APPLY: finished
1215 08:59:07.927731 (+ 419us) log.cc:844] Serialized 71 byte log entry
1215 08:59:07.927903 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:07.928039 (+ 136us) write_op.cc:454] Released schema lock
1215 08:59:07.928095 (+ 56us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":1560885,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":917,"apply.run_cpu_time_us":2823,"apply.run_wall_time_us":3544,"num_ops":9,"prepare.queue_time_us":812609,"prepare.run_cpu_time_us":11239,"prepare.run_wall_time_us":288278,"raft.queue_time_us":881,"raft.run_cpu_time_us":918,"raft.run_wall_time_us":1040,"replication_time_us":201475,"thread_start_us":1262424,"threads_started":4,"wal-append.queue_time_us":65356,"wal-append.run_cpu_time_us":3018,"wal-append.run_wall_time_us":1008793}]]}
I20251215 08:59:10.576982 22903 consensus_queue.cc:237] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:10.652688 23020 log.cc:927] Time spent T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20251215 08:59:10.656142 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 82440.8us
W20251215 08:59:10.658532 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 696507us
I20251215 08:59:10.662274 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:10.656595 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 846398us
I20251215 08:59:10.665602 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.662607 21304 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:10.666663 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1336) took 1360 ms. Trace:
I20251215 08:59:10.667083 21340 rpcz_store.cc:276] 1215 08:59:09.305966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.827757 (+521791us) service_pool.cc:225] Handling call
1215 08:59:09.828056 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:09.828262 (+ 206us) raft_consensus.cc:1534] Early marking committed up to index 5
1215 08:59:10.101297 (+273035us) log.cc:844] Serialized 14 byte log entry
1215 08:59:10.666353 (+565056us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:10.666405 (+ 52us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:10.666445 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.666512 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:10.666615 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":301822,"prepare.queue_time_us":361448,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":2992,"thread_start_us":534578,"threads_started":2,"wal-append.queue_time_us":390340}
I20251215 08:59:10.673115 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.662571 21262 raft_consensus.cc:1240] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:10.661511 22960 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:10.621344 22799 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.619372 23019 consensus_queue.cc:1059] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20251215 08:59:10.677111 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:10.677810 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:10.702054 21259 raft_consensus.cc:2393] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:10.679139 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:10.679893 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.680534 21407 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:10.681284 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.682017 23027 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.667944 22926 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.682466 23004 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:10.687779 22963 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.719013 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f1c639a483f4827997eae6f3a6e375d with delete_type TABLET_DATA_TOMBSTONED (TS 0a2fcb89589c401d8efc8d86da4ba0f4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49730
I20251215 08:59:10.718796 22963 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.658675 22927 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:10.721903 23004 raft_consensus.cc:3055] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:10.722244 23004 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:10.722994 23004 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.724135 23004 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:10.691078 22960 raft_consensus.cc:697] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:10.725919 22960 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.658332 22912 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.674795 22958 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.729527 22606 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:10.729753 22958 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.731864 22958 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.692770 22742 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:10.693189 22972 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:10.693768 21304 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:10.733579 22972 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.734115 22972 consensus_queue.cc:1059] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:10.674438 22921 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.693794 22959 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:10.735225 22959 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.735607 22959 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:10.735997 22837 raft_consensus.cc:3055] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:10.735745 22926 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.736203 22742 raft_consensus.cc:3055] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:10.736644 22742 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:10.695365 21973 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:10.695973 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.697777 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1439) took 1049 ms. Trace:
I20251215 08:59:10.746219 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.746774 21977 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:10.698535 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.711908 21229 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:10.620460 21411 rpcz_store.cc:269] 1215 08:59:08.791401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:08.792003 (+ 602us) service_pool.cc:225] Handling call
1215 08:59:10.429717 (+1637714us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:10.593565 (+163848us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3013888}
I20251215 08:59:10.656774 22609 consensus_queue.cc:1059] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20251215 08:59:10.716244 23027 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.731796 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:10.764721 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 48756.8us
I20251215 08:59:10.707798 22876 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:10.692456 21307 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.741914 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.663796 22919 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.768962 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:10.769667 21265 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:10.745321 21409 rpcz_store.cc:276] 1215 08:59:09.648440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.652391 (+ 3951us) service_pool.cc:225] Handling call
1215 08:59:09.652668 (+ 277us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:09.653051 (+ 383us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:09.653078 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:10.218215 (+565137us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b2c00092518. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:10.218618 (+ 403us) log.cc:844] Serialized 26 byte log entry
1215 08:59:10.527564 (+308946us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:59:10.527616 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:10.527708 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:10.697524 (+169816us) raft_consensus.cc:1685] finished
1215 08:59:10.697544 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:10.697725 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12,"raft.queue_time_us":437,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":69,"spinlock_wait_cycles":5837056,"thread_start_us":308555,"threads_started":1,"wal-append.queue_time_us":360606}
W20251215 08:59:10.759436 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 47588.5us
I20251215 08:59:10.750845 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:10.738147 22837 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:10.776079 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 83692.3us
I20251215 08:59:10.771783 21336 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:10.777415 22963 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.778122 22963 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:10.768607 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:10.773420 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.780920 21339 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:10.781600 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:10.778645 22993 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.783972 22609 raft_consensus.cc:3055] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:10.784433 22609 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:10.785260 22609 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.786484 22609 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:10.789780 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 49366.8us
I20251215 08:59:10.737748 22926 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.791692 22837 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.792699 23015 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:10.792862 22837 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:10.749293 21972 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:10.793373 21266 raft_consensus.cc:1240] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:10.717221 22910 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20251215 08:59:10.759104 22912 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.794952 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.811832 22963 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.823374 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:10.823796 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 58110.5us
I20251215 08:59:10.805483 22606 consensus_queue.cc:1059] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20251215 08:59:10.824563 22606 consensus_queue.cc:1243] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer 0a2fcb89589c401d8efc8d86da4ba0f4 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251215 08:59:10.795228 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:10.795892 21414 raft_consensus.cc:1275] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:10.797457 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.798300 21409 raft_consensus.cc:3055] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:10.830178 21409 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:10.795823 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:10.808247 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:10.810801 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:10.737465 22742 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.811305 21977 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:10.794559 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:10.767645 22902 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.818908 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:10.778877 23027 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:10.839773 21409 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.823587 22963 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.832084 21264 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:10.793861 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:10.832492 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:10.779008 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:10.841660 21258 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:10.873485 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:10.779366 22919 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.857790 21339 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:10.860438 21409 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:10.806469 22926 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.774920 22876 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.840569 22921 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:10.913223 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:10.921015 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 113842us
I20251215 08:59:10.783929 22993 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.893839 22940 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:10.935672 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:10.836462 22912 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.869818 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:10.916625 21974 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:10.834973 22895 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.097s
I20251215 08:59:10.836843 22959 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:10.915010 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:10.961076 21381 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:10.900451 22733 consensus_queue.cc:1059] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20251215 08:59:10.976125 21415 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:10.985739 21977 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:10.994166 21337 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:10.999331 21408 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:10.937350 23014 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:11.006153 21337 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:11.011482 21337 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:11.016233 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 55262.5us
I20251215 08:59:11.020098 21260 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:11.021212 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.022027 21264 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:11.024477 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.025254 21261 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:10.920905 22926 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:10.947974 22730 consensus_queue.cc:1059] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20251215 08:59:11.009290 22799 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:11.012931 22958 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.027570 23031 tablet_replica.cc:333] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: stopping tablet replica
I20251215 08:59:11.013737 22903 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20251215 08:59:10.958802 22940 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.103539 21980 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:11.027719 23014 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.161754 22896 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20251215 08:59:11.152033 21381 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:11.089849 22950 consensus_queue.cc:1059] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20251215 08:59:11.115270 21337 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.092866 22958 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.322881 21337 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:11.328195 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 314721us
I20251215 08:59:11.328271 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.329675 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:11.351745 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:11.354930 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:11.357879 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.359061 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.243333 22606 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:11.241386 22674 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:11.370972 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.364620 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.162492 22733 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.164s
I20251215 08:59:11.372743 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.313215 22963 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.384924 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.321247 22919 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:11.388674 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 149107us
I20251215 08:59:11.395222 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.350359 22960 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:11.399466 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.350585 22876 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.359342 22902 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:11.439435 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:11.440852 21409 consensus_meta.cc:287] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.546s user 0.006s sys 0.000s
I20251215 08:59:11.444141 21409 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:11.396183 22959 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.396667 22963 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:11.405637 22938 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:11.452430 21230 leader_election.cc:336] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.533s (SENT)
W20251215 08:59:11.457751 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:11.462767 21260 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:11.463948 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.464336 21265 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:11.465103 21259 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:59:11.465168 21264 raft_consensus.cc:1275] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:11.405809 22993 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.465483 21261 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251215 08:59:11.477675 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.477909 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:11.478441 21264 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:11.414527 22912 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.413648 22960 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:11.466507 23033 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20251215 08:59:11.500646 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:11.497923 22960 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:11.454342 22938 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:11.386057 22674 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.488940 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.493440 21264 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:11.486122 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.465442 22926 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:11.503873 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 150473us
W20251215 08:59:11.527035 23042 log.cc:927] Time spent T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20251215 08:59:11.528296 22958 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.524217 22938 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:11.531977 23004 consensus_meta.cc:287] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.807s user 0.010s sys 0.000s
I20251215 08:59:11.534763 22936 raft_consensus.cc:3055] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:11.535344 22936 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:11.536324 22920 raft_consensus.cc:3055] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:11.536767 22920 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:11.537808 22925 raft_consensus.cc:3055] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:11.537658 22920 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.538353 22925 raft_consensus.cc:740] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:11.538715 22920 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:11.539439 22925 consensus_queue.cc:260] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:11.518077 23040 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.279s user 0.001s sys 0.000s
I20251215 08:59:11.518810 22674 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:11.536473 22936 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.544576 22936 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:11.544878 22578 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20251215 08:59:11.496686 23008 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20251215 08:59:11.525334 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:11.535478 22927 consensus_meta.cc:287] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.814s user 0.009s sys 0.001s
I20251215 08:59:11.522998 22977 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.545687 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.549549 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.587822 21337 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:11.546805 21379 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:11.559293 21944 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.559365 21230 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.449098 22902 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.586189 21339 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:11.506260 23032 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
W20251215 08:59:11.506058 23036 log.cc:927] Time spent T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.450s user 0.001s sys 0.000s
I20251215 08:59:11.497098 23046 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.464439 22963 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:11.544976 22837 consensus_meta.cc:287] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.735s user 0.007s sys 0.000s
I20251215 08:59:11.503300 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.583945 22940 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.584228 23050 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.614290 22853 raft_consensus.cc:3055] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:11.638935 23050 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.627880 22912 consensus_queue.cc:1048] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20251215 08:59:11.623660 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:11.599839 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 53314.8us
I20251215 08:59:11.640385 22940 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:11.614413 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 55120.1us
W20251215 08:59:11.640935 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:11.599197 22958 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.642697 22958 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.643214 21944 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.644402 22831 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:11.644785 22831 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.637380 22963 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.595185 22977 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.644397 22940 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
I20251215 08:59:11.650593 22977 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:11.631654 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 479716us
I20251215 08:59:11.593578 23004 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:11.653359 21381 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:11.654073 22837 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:11.654489 22837 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:11.654879 22837 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:11.600322 23046 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.639444 22853 raft_consensus.cc:740] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:11.656293 22853 consensus_queue.cc:260] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.657619 22853 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:11.641497 21230 leader_election.cc:336] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.728s (SENT)
W20251215 08:59:11.602785 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:11.660476 21230 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.661321 23048 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:11.662420 23014 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.662984 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.664495 21336 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:11.666364 21339 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:11.667297 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.636583 22955 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:11.668824 21339 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:11.675189 21304 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:11.643437 23050 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.690410 21304 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:11.691737 21379 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:11.682179 21339 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20251215 08:59:11.669965 21972 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 357) took 1917 ms (client timeout 1882 ms). Trace:
I20251215 08:59:11.597306 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.668017 22938 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.644778 22958 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.683131 21971 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251215 08:59:11.648841 23038 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20251215 08:59:11.645362 22831 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:11.685426 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.685693 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:11.707163 21980 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:11.645165 22912 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20251215 08:59:11.648854 22961 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251215 08:59:11.677171 21974 consensus_meta.cc:287] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.677s user 0.007s sys 0.003s
I20251215 08:59:11.697269 21337 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:11.594528 22927 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.685930 22963 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:11.687577 23048 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:11.707531 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:11.721155 21978 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:11.710101 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:11.722227 21229 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:11.703399 22609 consensus_meta.cc:287] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.917s user 0.011s sys 0.000s
I20251215 08:59:11.687844 22955 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:11.698629 23051 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.697947 21336 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:11.715401 23058 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:11.697738 22940 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:11.724426 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:11.726226 21339 raft_consensus.cc:1240] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:11.709628 22958 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20251215 08:59:11.709254 22926 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:11.741417 23046 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.741348 22926 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:11.694779 21972 rpcz_store.cc:269] 1215 08:59:09.752825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:09.755858 (+ 3033us) service_pool.cc:225] Handling call
1215 08:59:10.440273 (+684415us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:11.669735 (+1229462us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b540066b2d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:11.669881 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69510144}
I20251215 08:59:11.743214 22926 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.735574 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:11.727149 23048 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.745515 22927 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.738773 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:11.730044 22955 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:11.733134 22963 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20251215 08:59:11.724509 21975 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:11.747793 21307 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:11.748219 21978 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:11.750756 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:11.744019 22837 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.716118 21974 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:11.763466 21977 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251215 08:59:11.747004 21413 raft_consensus.cc:606] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:11.753772 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.755654 23048 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.728633 23062 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:11.760192 21304 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:11.760985 21979 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:11.765421 21944 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:11.766656 22876 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:11.776566 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.777401 21265 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:11.777293 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 115638us
W20251215 08:59:11.777927 21230 leader_election.cc:336] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.793s (SENT)
I20251215 08:59:11.778379 21230 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.783825 21230 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:11.789309 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.766415 23062 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.753904 22609 consensus_queue.cc:1048] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20251215 08:59:11.772497 22916 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:11.774616 21333 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:11.794754 21230 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.769519 23058 raft_consensus.cc:697] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:11.772794 21974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46170 (request call id 307) took 1029 ms. Trace:
I20251215 08:59:11.801410 21333 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:11.734442 23051 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.794396 23063 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:11.792912 23048 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:11.819559 21257 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:11.770965 23004 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20251215 08:59:11.822887 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:11.810474 21230 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.826786 21336 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:11.793334 22876 raft_consensus.cc:697] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:11.827320 21257 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:11.827777 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.802053 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.776399 22940 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20251215 08:59:11.843691 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.823458 21413 raft_consensus.cc:606] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:11.845258 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:11.826936 23030 log.cc:927] Time spent T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.823s user 0.002s sys 0.002s
I20251215 08:59:11.829401 21230 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:11.828819 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:11.818884 22606 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
W20251215 08:59:11.830400 23034 log.cc:927] Time spent T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20251215 08:59:11.825304 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:11.833118 23048 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:11.859285 23055 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20251215 08:59:11.795612 23053 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:11.855369 22606 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:11.824898 22916 raft_consensus.cc:697] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:11.863416 23008 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:11.863886 22916 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.797600 23062 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:11.828557 22939 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251215 08:59:11.880529 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:11.881400 21412 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:11.875546 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.876767 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.878963 21413 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:11.852943 22963 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:11.886579 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:11.887622 21978 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:11.889914 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:11.856120 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:11.893978 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:11.827924 22876 consensus_queue.cc:237] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:11.802019 21974 rpcz_store.cc:276] 1215 08:59:10.741140 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.745293 (+ 4153us) service_pool.cc:225] Handling call
1215 08:59:11.770618 (+1025325us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:11.899880 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:11.900131 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:11.898649 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 69332.4us
I20251215 08:59:11.901360 21409 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:11.902163 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.902946 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.826967 23058 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.882222 22938 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.908867 21230 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.880138 22609 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:11.922140 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.922202 22609 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } attrs { replace: true } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.917869 21304 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:11.913314 22939 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:11.885979 22958 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20251215 08:59:11.924360 23063 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:11.925374 22958 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:11.925464 22609 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:11.925727 22939 raft_consensus.cc:697] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:11.925925 22958 raft_consensus.cc:697] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:11.905961 21257 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:11.926913 22958 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.927467 22939 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.929981 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:11.919680 21230 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.921320 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.918279 21972 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:11.952589 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:11.936060 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:11.878584 22730 consensus_queue.cc:1059] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20251215 08:59:11.941992 23051 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:11.943207 23063 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:11.945503 22955 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:11.946547 22925 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:11.955700 21230 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:11.963819 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.964469 21415 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:11.965392 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.966415 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.917392 22837 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:11.978211 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:11.980131 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:11.988694 23060 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251215 08:59:11.991176 23060 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:59:11.994691 21407 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:11.960902 23034 log.cc:927] Time spent T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251215 08:59:11.999938 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:12.001427 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:11.958273 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:11.974797 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:12.004540 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.014362 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.016238 21408 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251215 08:59:12.012928 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.004194 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:11.962204 23063 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:11.966213 23062 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:11.958498 22920 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20251215 08:59:12.009606 22919 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:11.961930 22961 consensus_queue.cc:1059] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20251215 08:59:12.028576 21414 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:11.975028 23037 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:12.031682 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:12.012933 23035 raft_consensus.cc:3055] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:12.008972 22972 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:12.043565 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1344) took 1289 ms. Trace:
I20251215 08:59:12.014470 22609 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:11.979597 23051 consensus_queue.cc:1059] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20251215 08:59:12.047444 21307 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:12.029803 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:12.059006 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.048758 21261 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:12.053723 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:12.063548 21266 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
W20251215 08:59:12.056438 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 170114us
I20251215 08:59:12.066004 21265 raft_consensus.cc:1240] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:12.055152 21378 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:12.022258 23063 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:12.052611 21335 rpcz_store.cc:276] 1215 08:59:10.754386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.758395 (+ 4009us) service_pool.cc:225] Handling call
1215 08:59:10.758676 (+ 281us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:10.775657 (+ 16981us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:10.775685 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:10.776052 (+ 367us) log.cc:844] Serialized 26 byte log entry
1215 08:59:11.111746 (+335694us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:11.234569 (+122823us) log.cc:844] Serialized 14 byte log entry
1215 08:59:11.234870 (+ 301us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:11.234967 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:11.827703 (+592736us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:12.029596 (+201893us) raft_consensus.cc:1685] finished
1215 08:59:12.043302 (+ 13706us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:12.043495 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16647,"prepare.queue_time_us":123201,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":4865,"raft.queue_time_us":24444,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":72057,"spinlock_wait_cycles":4996352,"thread_start_us":290369,"threads_started":1,"wal-append.queue_time_us":866737}
I20251215 08:59:12.042336 23044 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:12.081825 21266 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:12.082530 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:12.088486 21266 raft_consensus.cc:606] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:12.088879 21266 raft_consensus.cc:614] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:12.091540 22943 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:12.019949 21410 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:12.003957 22960 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.086690 21381 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:12.095752 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:12.104606 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:12.076838 22938 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:12.083089 23035 raft_consensus.cc:740] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:12.084862 22919 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:12.061309 21257 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:12.099891 22609 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:12.109989 23044 raft_consensus.cc:2764] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251215 08:59:12.133057 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:12.144203 21266 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.093564 23046 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:12.086761 23037 raft_consensus.cc:697] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:12.109992 21307 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:12.135239 21338 raft_consensus.cc:2393] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:12.057260 22606 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.113437 22916 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:12.147948 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 92874.5us
I20251215 08:59:12.149121 21307 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:12.122885 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 153520us
I20251215 08:59:12.134315 22922 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:12.154568 22922 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.138988 21408 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251215 08:59:12.134120 23078 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:12.161566 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:12.161868 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:12.134969 22938 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.164084 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:12.165781 21414 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:12.155376 22609 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:12.172703 21381 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:12.172233 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:12.166821 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:12.158198 23044 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:12.135152 22831 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.164378 22853 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:12.166215 23078 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:12.181583 22609 consensus_queue.cc:1059] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:12.182304 22938 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.182888 21408 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:12.168721 22955 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:12.183765 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:12.203243 21230 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:12.190891 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:12.160948 22935 consensus_queue.cc:1059] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20251215 08:59:12.204574 21230 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:12.212435 21978 raft_consensus.cc:1240] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:12.177196 23053 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:12.240141 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 49363.7us
I20251215 08:59:12.238783 21266 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:12.247464 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:12.250689 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:12.179158 23061 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:12.180286 22853 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.225087 23004 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:12.272173 21266 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251215 08:59:12.231853 23062 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.178349 22916 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.069s
I20251215 08:59:12.172204 23035 consensus_queue.cc:260] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.184029 23051 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:12.180913 23078 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.245819 22939 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:12.237556 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:12.278566 23059 raft_consensus.cc:3055] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:12.270664 22606 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:12.246127 22958 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:12.277453 22960 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:12.281662 23072 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20251215 08:59:12.283034 23059 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:12.290728 23082 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:12.290957 22978 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:12.291989 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 371) took 1553 ms. Trace:
I20251215 08:59:12.289180 23035 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.277201 23004 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.297760 23059 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.299033 23004 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.277370 22972 consensus_queue.cc:1059] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20251215 08:59:12.283345 23061 raft_consensus.cc:697] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:12.301739 21264 raft_consensus.cc:1240] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:12.211867 23037 consensus_queue.cc:237] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.295614 22960 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:12.253152 23053 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.300662 22922 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:12.282205 23051 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.320430 21264 raft_consensus.cc:1240] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:12.321635 21972 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:12.328671 21264 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251215 08:59:12.308671 21974 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
W20251215 08:59:12.335512 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.341514 21332 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:12.344827 22853 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.347013 22902 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.352692 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:12.352833 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:12.353502 21976 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.354502 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:12.355749 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:12.331518 22831 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.358394 21229 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:12.363587 23080 log.cc:927] Time spent T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.385s user 0.003s sys 0.001s
I20251215 08:59:12.312552 22960 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.365463 23051 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.295811 21971 rpcz_store.cc:276] 1215 08:59:10.738816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.744997 (+ 6181us) service_pool.cc:225] Handling call
1215 08:59:11.614339 (+869342us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:11.680888 (+ 66549us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:11.707264 (+ 26376us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:12.291782 (+584518us) spinlock_profiling.cc:243] Waited 50.5 ms on lock 0x7b540066b2d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:12.291931 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":131051264}
I20251215 08:59:12.368053 22950 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:12.371057 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:12.371842 21971 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
W20251215 08:59:12.374032 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:12.375013 23076 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
W20251215 08:59:12.375105 23088 log.cc:927] Time spent T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251215 08:59:12.385447 23078 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.388032 23078 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:12.390058 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 143249us
I20251215 08:59:12.391829 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:12.397377 21260 raft_consensus.cc:3055] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:12.397830 21260 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:12.398828 21260 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.400076 21260 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.402542 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:12.403604 21264 raft_consensus.cc:2393] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:12.410236 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:12.414363 23103 consensus_queue.cc:1059] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20251215 08:59:12.426594 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 315545us
I20251215 08:59:12.381393 22955 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20251215 08:59:12.432039 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.362357 23061 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.447458 21260 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20251215 08:59:12.447944 21260 pending_rounds.cc:85] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 5
I20251215 08:59:12.448225 21260 pending_rounds.cc:107] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251215 08:59:12.448580 21260 raft_consensus.cc:2889] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:12.426632 22960 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.454102 21232 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:12.441550 22916 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:12.459110 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:12.460223 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:12.460860 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.464246 22940 raft_consensus.cc:3055] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:12.464818 22940 raft_consensus.cc:740] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:12.466051 23059 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:12.465842 22940 consensus_queue.cc:260] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.451767 22958 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:12.474531 22939 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:12.458355 22960 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:12.467435 22940 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.477895 22831 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:12.486155 22925 consensus_meta.cc:287] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.518s user 0.012s sys 0.000s
I20251215 08:59:12.505237 21381 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:12.503852 23053 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.487262 22958 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.476241 22916 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.517612 22831 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.555461 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1351) took 1714 ms. Trace:
I20251215 08:59:12.555923 21334 rpcz_store.cc:276] 1215 08:59:10.841428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.851301 (+ 9873us) service_pool.cc:225] Handling call
1215 08:59:10.924077 (+ 72776us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:11.025296 (+101219us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:11.025342 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:11.025404 (+ 62us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:11.025450 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:12.551687 (+1526237us) spinlock_profiling.cc:243] Waited 67.7 ms on lock 0x7b5400123ed4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:12.555251 (+ 3564us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:12.555394 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155758848}
I20251215 08:59:12.557701 22831 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.494119 22960 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.559038 21976 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:12.547269 22925 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:12.559514 22916 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.559968 22925 raft_consensus.cc:697] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:12.560453 21980 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251215 08:59:12.560861 22925 consensus_queue.cc:237] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:12.564846 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 115962us
W20251215 08:59:12.566457 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.573980 23094 consensus_queue.cc:1059] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20251215 08:59:12.284145 23046 consensus_queue.cc:1175] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "beefe15185ac42ef9a20251007f39a56" responder_term: 4 status { last_received { term: 3 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251215 08:59:12.576727 22916 raft_consensus.cc:3055] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:12.577209 22916 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:12.575284 22853 consensus_queue.cc:1059] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20251215 08:59:12.559556 22960 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.587849 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:12.588519 21412 raft_consensus.cc:3055] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:12.588788 21412 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:12.589509 21412 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.590407 21412 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.592382 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.599221 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:12.585192 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.605712 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.606925 21410 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20251215 08:59:12.608877 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.608907 23107 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:12.609491 23099 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
W20251215 08:59:12.611150 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.554407 23063 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.578069 22916 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.619894 21412 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:12.614280 21408 raft_consensus.cc:606] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:12.630681 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:12.609719 23107 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.626375 22674 raft_consensus.cc:3055] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:12.637501 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:12.638252 22674 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:12.635128 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.614796 23103 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20251215 08:59:12.629264 22916 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:12.635800 22960 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:12.629281 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.629659 23055 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:12.663165 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:12.663600 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.635854 23106 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:12.671021 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.671861 21412 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
W20251215 08:59:12.672307 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:12.673708 23115 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:12.675479 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:12.669309 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:12.679661 22951 raft_consensus.cc:3055] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:12.681082 23008 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:12.681708 22951 raft_consensus.cc:740] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:12.683800 23008 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:12.687594 23110 log.cc:927] Time spent T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20251215 08:59:12.688903 21971 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20251215 08:59:12.689234 23063 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.689494 21971 pending_rounds.cc:85] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Aborting all ops after (but not including) 5
I20251215 08:59:12.689877 21971 pending_rounds.cc:107] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20251215 08:59:12.690207 23103 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:12.681098 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:12.679212 23035 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:12.696952 21946 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:12.667910 21410 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:12.678499 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:12.652454 22960 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.690218 21971 raft_consensus.cc:2889] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:12.698664 23004 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:12.697182 22922 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:12.684221 23008 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:12.703760 23107 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.674389 23115 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.710752 21412 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20251215 08:59:12.680745 23055 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.709900 22958 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.724020 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:12.719465 22960 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.675207 22674 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.725178 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:12.726599 21258 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:12.727159 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 1400) took 1009 ms. Trace:
I20251215 08:59:12.727629 21258 rpcz_store.cc:276] 1215 08:59:11.717708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.717815 (+ 107us) service_pool.cc:225] Handling call
1215 08:59:12.724981 (+1007166us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:12.727084 (+ 2103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42711936}
I20251215 08:59:12.726655 22958 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:12.730100 22958 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.732244 22958 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:12.701730 23035 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.722807 23059 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20251215 08:59:12.682080 23106 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.691515 22951 consensus_queue.cc:260] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:12.723157 22922 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:12.815843 23115 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:12.829783 21977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 424) took 1746 ms. Trace:
I20251215 08:59:12.830390 21977 rpcz_store.cc:276] 1215 08:59:11.083584 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.083741 (+ 157us) service_pool.cc:225] Handling call
1215 08:59:11.763239 (+679498us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:11.763896 (+ 657us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:12.829464 (+1065568us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b540068d650. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:12.829684 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1551039232}
W20251215 08:59:12.841518 22766 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20251215 08:59:13.325361 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:13.302836 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:13.322957 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:13.333662 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:13.312551 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:13.321739 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:13.322577 21410 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20251215 08:59:13.305919 21304 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.029s sys 0.019s
W20251215 08:59:13.374449 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:13.342258 22660 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:13.381271 22674 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:13.321409 21411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.010s sys 0.014s
W20251215 08:59:13.309746 21307 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.034s sys 0.072s
I20251215 08:59:13.331694 21412 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251215 08:59:12.886363 22935 raft_consensus.cc:993] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:13.379770 22831 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:13.404407 21979 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.004s sys 0.002s
W20251215 08:59:13.404973 21979 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.004s sys 0.002s
W20251215 08:59:13.405350 23103 consensus_meta.cc:287] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.715s user 0.014s sys 0.012s
I20251215 08:59:13.370232 22951 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:13.406208 23103 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:12.879281 22961 raft_consensus.cc:3055] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Stepping down as leader of term 2
W20251215 08:59:12.860323 23069 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.971s user 0.001s sys 0.002s
I20251215 08:59:12.924288 23027 raft_consensus.cc:3055] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
W20251215 08:59:13.410692 21307 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s user 0.039s sys 0.076s
W20251215 08:59:12.866575 22896 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20251215 08:59:13.401636 21304 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s user 0.039s sys 0.024s
I20251215 08:59:12.860244 22922 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:12.905572 21975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 419) took 2229 ms (client timeout 1861 ms). Trace:
I20251215 08:59:13.416610 22912 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:13.417264 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 963051us
I20251215 08:59:13.417616 23053 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:13.412276 22961 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:13.427412 22912 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.037616 23008 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:13.428373 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 974613us
I20251215 08:59:13.430732 22912 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.432351 21229 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:13.431957 23053 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:13.413911 21411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.017s sys 0.016s
I20251215 08:59:13.432737 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1483) took 2068 ms. Trace:
I20251215 08:59:13.428575 22961 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.443293 23051 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:13.405243 22831 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:13.452693 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:13.452567 22935 raft_consensus.cc:1081] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:13.453651 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1317) took 1057 ms. Trace:
I20251215 08:59:13.438544 21341 rpcz_store.cc:276] 1215 08:59:11.364298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.711879 (+347581us) service_pool.cc:225] Handling call
1215 08:59:11.724233 (+ 12354us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:11.728807 (+ 4574us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:11.728851 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:11.738668 (+ 9817us) log.cc:844] Serialized 26 byte log entry
1215 08:59:11.838190 (+ 99522us) spinlock_profiling.cc:243] Waited 9.73 ms on lock 0x7b48001eda58. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398cacde21 00007f398cac4eb7 00007f398cac4d4a 00007f398cbbbf7f 00007f398cbbbc47 00007f398cbd026c 00007f398cbd0218 00007f398cbd0198 00007f398cbd0153 00007f398cbcf3fc 00007f398bb0a516
1215 08:59:11.895754 (+ 57564us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:59:11.895808 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:11.895896 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:12.859639 (+963743us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:13.432125 (+572486us) raft_consensus.cc:1685] finished
1215 08:59:13.432189 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.432466 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4176,"raft.queue_time_us":5613,"raft.run_cpu_time_us":2116,"raft.run_wall_time_us":105829,"spinlock_wait_cycles":76562048,"thread_start_us":156713,"threads_started":1,"wal-append.queue_time_us":150123}
I20251215 08:59:13.455806 22902 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:13.456123 23106 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.457031 22895 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:13.424293 21975 rpcz_store.cc:269] 1215 08:59:10.675680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.684020 (+ 8340us) service_pool.cc:225] Handling call
1215 08:59:12.905351 (+2221331us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54004e1954. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:12.905485 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1774175872}
I20251215 08:59:13.461702 22895 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:13.459432 21410 rpcz_store.cc:276] 1215 08:59:12.396423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:12.727713 (+331290us) service_pool.cc:225] Handling call
1215 08:59:12.775452 (+ 47739us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:13.413352 (+637900us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.453458 (+ 40106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:13.470053 23027 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:13.466244 22902 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.478039 22903 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:13.478869 22903 raft_consensus.cc:697] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:13.479372 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:13.472376 22895 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:13.481103 22920 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:13.482918 22945 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.484926 22938 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.485504 22945 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.485684 22938 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.486022 22920 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.488041 22920 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:13.488127 22945 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:13.489912 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:13.480536 22903 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.489614 22938 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:13.473155 23051 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.491212 22896 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:13.504549 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 867074us
I20251215 08:59:13.500851 22902 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.508613 22961 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:13.509194 23060 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.510607 23035 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.523196 22920 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:13.526338 23055 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.528369 22935 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.537313 22831 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.538231 21257 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:13.539258 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.531926 23060 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.539726 23051 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.541632 22902 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:13.543358 21230 leader_election.cc:336] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.965s (SENT)
W20251215 08:59:13.544111 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:13.539258 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:13.550833 21409 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1300) took 1643 ms (client timeout 1944 ms). Trace:
I20251215 08:59:13.548223 22902 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.549119 21230 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:13.553131 22935 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.553885 22976 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:13.555614 22935 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:13.551929 21409 rpcz_store.cc:269] 1215 08:59:11.907083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.907302 (+ 219us) service_pool.cc:225] Handling call
1215 08:59:13.538859 (+1631557us) spinlock_profiling.cc:243] Waited 9.31 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:13.550717 (+ 11858us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21405440}
I20251215 08:59:13.556015 22977 consensus_queue.cc:1059] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20251215 08:59:13.561678 22977 raft_consensus.cc:3055] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:13.562659 23037 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:13.562718 22977 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:13.563478 23037 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.564100 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1404) took 1832 ms. Trace:
I20251215 08:59:13.565362 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1479) took 1054 ms. Trace:
I20251215 08:59:13.564112 22977 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.535845 22896 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.561777 22902 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.566862 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.565636 21412 rpcz_store.cc:276] 1215 08:59:12.505611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:13.500995 (+995384us) service_pool.cc:225] Handling call
1215 08:59:13.525717 (+ 24722us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:13.556277 (+ 30560us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:13.556305 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:13.558023 (+ 1718us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:13.560185 (+ 2162us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.560247 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.560330 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:13.570199 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.564532 21259 rpcz_store.cc:276] 1215 08:59:11.731633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.731814 (+ 181us) service_pool.cc:225] Handling call
1215 08:59:13.563059 (+1831245us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:13.563348 (+ 289us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:13.563548 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:13.563575 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:13.563611 (+ 36us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:13.563692 (+ 81us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.563782 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.563906 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44989696}
I20251215 08:59:13.569662 23035 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:13.572681 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.573259 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:13.573798 21260 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20251215 08:59:13.573711 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:13.575316 21974 raft_consensus.cc:2410] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251215 08:59:13.558159 22938 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:13.569674 22976 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:13.571988 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:13.565800 23053 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.570163 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1482) took 1029 ms. Trace:
I20251215 08:59:13.559113 22920 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.584697 23055 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:13.591828 23035 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.592566 23055 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.584975 23037 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.565084 22935 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:13.590683 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.546494 23027 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.581374 23123 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.593353 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:13.607029 21262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1281) took 1871 ms (client timeout 1864 ms). Trace:
I20251215 08:59:13.596434 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34992 (request call id 15) took 1420 ms. Trace:
W20251215 08:59:13.607571 21262 rpcz_store.cc:269] 1215 08:59:11.736098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.737912 (+ 1814us) service_pool.cc:225] Handling call
1215 08:59:13.590285 (+1852373us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:13.606948 (+ 16663us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29048576}
I20251215 08:59:13.608592 23129 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:13.609247 23103 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:13.607859 21408 rpcz_store.cc:276] 1215 08:59:12.172444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:12.182721 (+ 10277us) service_pool.cc:225] Handling call
1215 08:59:13.593075 (+1410354us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b540026ded4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bae4d3 00007f398cb6c13a 00007f3986107934 00007f39861078c2 00007f39861077f1 00007f398610776c 00007f3986106a22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:13.593247 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":910798208}
I20251215 08:59:13.610270 23098 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:13.611114 23098 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:13.611588 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 54739.4us
I20251215 08:59:13.611977 22976 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.614626 21338 raft_consensus.cc:1240] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:13.599326 22920 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:13.594633 22938 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:13.620760 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 85603.4us
I20251215 08:59:13.614063 23098 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.614876 22976 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:13.624150 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 76756.8us
I20251215 08:59:13.629154 21409 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:13.621822 23027 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:13.588392 21410 rpcz_store.cc:276] 1215 08:59:12.537016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:13.499842 (+962826us) service_pool.cc:225] Handling call
1215 08:59:13.518450 (+ 18608us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:13.547165 (+ 28715us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:13.547738 (+ 573us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:13.552734 (+ 4996us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:13.556534 (+ 3800us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.558100 (+ 1566us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.565883 (+ 7783us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:13.630293 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1319) took 1062 ms. Trace:
I20251215 08:59:13.633359 21409 rpcz_store.cc:276] 1215 08:59:12.568604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:13.598794 (+1030190us) service_pool.cc:225] Handling call
1215 08:59:13.603474 (+ 4680us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:13.629443 (+ 25969us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.630244 (+ 801us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:13.635811 21977 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251215 08:59:13.636547 23055 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.637290 21341 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:13.638432 23008 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.641150 23055 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.641813 23055 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.644302 23055 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.644678 22896 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.641636 23123 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.653868 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:13.654281 21973 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:13.656741 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1469) took 1782 ms. Trace:
I20251215 08:59:13.625449 22935 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.657186 21407 rpcz_store.cc:276] 1215 08:59:11.874118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.874251 (+ 133us) service_pool.cc:225] Handling call
1215 08:59:11.874517 (+ 266us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:11.995220 (+120703us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.656224 (+1661004us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b540006ead4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:13.656549 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":275851520}
I20251215 08:59:13.647964 23060 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.658422 21307 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:13.647758 23123 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.665307 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1471) took 1790 ms. Trace:
I20251215 08:59:13.666303 22935 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:13.666841 23035 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.667266 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:13.668015 23055 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:13.669340 21411 rpcz_store.cc:276] 1215 08:59:11.874641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.877915 (+ 3274us) service_pool.cc:225] Handling call
1215 08:59:12.687871 (+809956us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:12.688038 (+ 167us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:12.695599 (+ 7561us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:12.698320 (+ 2721us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:12.699240 (+ 920us) log.cc:844] Serialized 26 byte log entry
1215 08:59:13.498816 (+799576us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:13.504202 (+ 5386us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.506732 (+ 2530us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:13.647078 (+140346us) spinlock_profiling.cc:243] Waited 8.88 ms on lock 0x7b18000d8400. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:13.658228 (+ 11150us) raft_consensus.cc:1685] finished
1215 08:59:13.659704 (+ 1476us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.665246 (+ 5542us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7669,"raft.queue_time_us":7364,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":403,"spinlock_wait_cycles":29041664,"thread_start_us":740405,"threads_started":1,"wal-append.queue_time_us":754565}
I20251215 08:59:13.671617 21409 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:13.675920 21409 raft_consensus.cc:606] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:13.676198 21409 raft_consensus.cc:614] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:13.676857 21378 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:13.676929 23105 consensus_queue.cc:1059] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:13.678087 22935 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:13.678432 23105 raft_consensus.cc:3055] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:13.678915 23105 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:13.680207 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1353) took 2811 ms. Trace:
I20251215 08:59:13.658403 22895 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.680564 21332 rpcz_store.cc:276] 1215 08:59:10.869184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.871318 (+ 2134us) service_pool.cc:225] Handling call
1215 08:59:12.338809 (+1467491us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b5800014198. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:12.341264 (+ 2455us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:12.362553 (+ 21289us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.679798 (+1317245us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b54001252d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:13.680021 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67937408}
I20251215 08:59:13.678582 22935 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:13.683288 21266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1414) took 1580 ms. Trace:
I20251215 08:59:13.686038 21266 rpcz_store.cc:276] 1215 08:59:12.103191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:12.295705 (+192514us) service_pool.cc:225] Handling call
1215 08:59:12.296010 (+ 305us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:12.337544 (+ 41534us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:12.337589 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:12.337639 (+ 50us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:12.337679 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.682783 (+1345104us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b5400358254. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:13.682851 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.683061 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93414784}
I20251215 08:59:13.687690 23060 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:13.687942 21260 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:13.688973 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.688602 21413 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251215 08:59:13.689563 21258 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:13.689587 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.689913 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 133) took 1774 ms. Trace:
I20251215 08:59:13.690496 21265 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:13.685369 22895 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.690421 21413 rpcz_store.cc:276] 1215 08:59:11.915737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.915922 (+ 185us) service_pool.cc:225] Handling call
1215 08:59:13.687837 (+1771915us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:13.688121 (+ 284us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:13.689692 (+ 1571us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.689871 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4830976}
I20251215 08:59:13.688514 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.689581 22935 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:13.692348 21264 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:13.680042 23105 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.703296 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1311) took 1754 ms. Trace:
I20251215 08:59:13.702598 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.707669 23060 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.708580 22977 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:13.709111 21263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1403) took 1969 ms. Trace:
I20251215 08:59:13.709600 21263 rpcz_store.cc:276] 1215 08:59:11.729686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.729861 (+ 175us) service_pool.cc:225] Handling call
1215 08:59:13.696442 (+1966581us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:13.696774 (+ 332us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:13.697090 (+ 316us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:13.697134 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:13.697212 (+ 78us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:13.697252 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.697333 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.698601 (+ 1268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44915072}
I20251215 08:59:13.703218 21262 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:13.704270 21414 rpcz_store.cc:276] 1215 08:59:11.948866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:12.048517 (+ 99651us) service_pool.cc:225] Handling call
1215 08:59:12.094384 (+ 45867us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:12.173491 (+ 79107us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.700293 (+1526802us) spinlock_profiling.cc:243] Waited 71.1 ms on lock 0x7b5400341f54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:13.703112 (+ 2819us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":163566976}
I20251215 08:59:13.711869 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:13.712095 22895 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.714816 21261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1402) took 1986 ms. Trace:
I20251215 08:59:13.715199 21261 rpcz_store.cc:276] 1215 08:59:11.728317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.728471 (+ 154us) service_pool.cc:225] Handling call
1215 08:59:11.842580 (+114109us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:11.842808 (+ 228us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:12.072394 (+229586us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.714321 (+1641927us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54002e54d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:13.714636 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":499010304}
I20251215 08:59:13.714603 22895 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.716872 21381 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:13.725997 23105 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:13.725891 22742 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:13.727835 22674 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:13.728678 22895 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.718258 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1484) took 1174 ms. Trace:
I20251215 08:59:13.731196 21412 rpcz_store.cc:276] 1215 08:59:12.543687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:13.577501 (+1033814us) service_pool.cc:225] Handling call
1215 08:59:13.599139 (+ 21638us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:13.676067 (+ 76928us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:13.676112 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:13.682140 (+ 6028us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:13.682185 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.718010 (+ 35825us) spinlock_profiling.cc:243] Waited 64.5 ms on lock 0x7b5400251054. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:13.718056 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.718184 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148272896}
I20251215 08:59:13.734741 21341 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:13.720630 21409 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:13.735898 21409 raft_consensus.cc:606] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:13.736344 21409 raft_consensus.cc:614] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:13.725905 21334 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
W20251215 08:59:13.717000 23130 log.cc:927] Time spent T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.120s user 0.002s sys 0.001s
I20251215 08:59:13.713133 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:13.740907 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:13.741767 23128 log.cc:927] Time spent T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.269s user 0.002s sys 0.000s
I20251215 08:59:13.742767 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251215 08:59:13.743031 21979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 394) took 1004 ms. Trace:
I20251215 08:59:13.743374 21979 rpcz_store.cc:276] 1215 08:59:12.738970 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:12.739132 (+ 162us) service_pool.cc:225] Handling call
1215 08:59:12.782422 (+ 43290us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:12.782850 (+ 428us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:12.782887 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:12.791180 (+ 8293us) log.cc:844] Serialized 26 byte log entry
1215 08:59:13.405756 (+614576us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:13.405875 (+ 119us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.406001 (+ 126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:13.740989 (+334988us) spinlock_profiling.cc:243] Waited 8.59 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:13.742700 (+ 1711us) raft_consensus.cc:1685] finished
1215 08:59:13.742726 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.742960 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7672,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":1780,"spinlock_wait_cycles":19753856,"thread_start_us":613885,"threads_started":1,"wal-append.queue_time_us":677098}
W20251215 08:59:13.752255 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 85103.5us
I20251215 08:59:13.754863 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:13.755929 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:13.757514 22895 consensus_queue.cc:1059] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:13.757813 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.757879 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.758495 21261 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:13.758798 21265 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:13.758522 22895 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:13.759349 22903 raft_consensus.cc:3055] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:13.759706 22903 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:13.760320 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:13.760622 22903 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.764046 22903 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:13.762382 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:13.763025 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:13.769236 22895 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.769536 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:13.769923 22895 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.772149 22895 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.772830 21381 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:13.772975 22730 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:13.774106 23014 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:13.774585 21381 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:13.775157 22730 consensus_queue.cc:1059] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:13.774597 23014 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:13.775904 23014 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:13.778184 21162 catalog_manager.cc:5654] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 1 to 2, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 2 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:13.779461 22935 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.782723 21974 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20251215 08:59:13.783246 21974 pending_rounds.cc:85] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Aborting all ops after (but not including) 4
I20251215 08:59:13.783550 21974 pending_rounds.cc:107] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251215 08:59:13.783943 21974 raft_consensus.cc:2889] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:13.734943 22938 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.785620 22853 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:13.786005 22853 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:13.786321 22853 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:13.787621 22925 raft_consensus.cc:3055] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:13.788079 22925 raft_consensus.cc:740] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:13.789141 22925 consensus_queue.cc:260] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.790091 22925 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:13.793615 21163 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45538 (request call id 150) took 11742 ms (client timeout 15000 ms). Trace:
I20251215 08:59:13.722572 23055 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:13.794721 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:13.795512 23055 consensus_queue.cc:1059] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:13.762415 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:13.797382 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 124933us
I20251215 08:59:13.798755 23098 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:13.799064 23060 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.804215 22950 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:13.810138 22935 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:13.811107 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 48849.5us
I20251215 08:59:13.813131 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:13.814095 21381 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:13.815374 22895 consensus_queue.cc:1059] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251215 08:59:13.794072 21163 rpcz_store.cc:269] 1215 08:59:02.022504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:02.026541 (+ 4037us) service_pool.cc:225] Handling call
1215 08:59:02.835077 (+808536us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400030cd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398d93b13c 00007f398da3544e 00007f398bf334a4 00007f398bf33432 00007f398bf33361 00007f398bf332dc 00007f398bf32592 00007f3985ef355c 00007f3985ef2996
1215 08:59:13.764890 (+10929813us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:10.870518 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:11.032368 (+161850us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:11.032412 (+ 44us) write_op.cc:435] Acquired schema lock
1215 08:59:11.032435 (+ 23us) tablet.cc:660] Decoding operations
1215 08:59:11.033170 (+ 735us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:11.033269 (+ 99us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:11.033313 (+ 44us) tablet.cc:683] Acquiring locks for 3 operations
1215 08:59:11.033736 (+ 423us) tablet.cc:699] Row locks acquired
1215 08:59:11.033761 (+ 25us) write_op.cc:260] PREPARE: finished
1215 08:59:11.033951 (+ 190us) write_op.cc:270] Start()
1215 08:59:11.034092 (+ 141us) write_op.cc:276] Timestamp: P: 1765789151033890 usec, L: 0
1215 08:59:11.034121 (+ 29us) op_driver.cc:348] REPLICATION: starting
1215 08:59:11.034999 (+ 878us) log.cc:844] Serialized 1302 byte log entry
1215 08:59:11.799905 (+764906us) op_driver.cc:464] REPLICATION: finished
1215 08:59:12.013521 (+213616us) write_op.cc:301] APPLY: starting
1215 08:59:12.013601 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:12.013731 (+ 130us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:12.013855 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:12.014306 (+ 451us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:12.014494 (+ 188us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1215 08:59:12.014543 (+ 49us) write_op.cc:312] APPLY: finished
1215 08:59:12.014738 (+ 195us) log.cc:844] Serialized 35 byte log entry
1215 08:59:12.014888 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:12.014976 (+ 88us) write_op.cc:454] Released schema lock
1215 08:59:12.015012 (+ 36us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:12.015349 (+ 337us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:12.015373 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2212387,"reported_tablets":7,"spinlock_wait_cycles":554842496,"child_traces":[["op",{"apply.queue_time_us":210455,"apply.run_cpu_time_us":1804,"apply.run_wall_time_us":1804,"num_ops":3,"prepare.queue_time_us":131345,"prepare.run_cpu_time_us":20165,"prepare.run_wall_time_us":490483,"raft.queue_time_us":242898,"raft.run_cpu_time_us":21708,"raft.run_wall_time_us":218266,"replication_time_us":740787,"thread_start_us":694064,"threads_started":4,"wal-append.queue_time_us":431032,"wal-append.run_cpu_time_us":17116,"wal-append.run_wall_time_us":1536670}]]}
I20251215 08:59:13.816396 22895 raft_consensus.cc:3055] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:13.817083 23027 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:13.814126 21975 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:13.818085 22730 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:13.818537 22730 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:13.816856 22895 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:13.818910 22730 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:13.821736 23031 raft_consensus.cc:2243] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 08:59:13.822520 23098 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.823060 23031 raft_consensus.cc:2272] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 08:59:13.811785 22853 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.823076 23098 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.825978 22853 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:13.825752 22895 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:13.828825 22895 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:13.826550 23098 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.831590 23048 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.832448 22730 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.832257 23048 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.833941 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:13.866194 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:13.866402 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:13.866730 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:13.866933 21412 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:13.867372 23048 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.868543 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:13.837637 23031 ts_tablet_manager.cc:1916] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251215 08:59:13.839087 21334 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:13.862736 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:13.839017 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:13.840348 22896 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.870528 21262 raft_consensus.cc:2410] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20251215 08:59:13.870458 22896 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.870653 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:13.872516 22896 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:13.841725 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:13.873286 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:13.841838 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:13.841876 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:13.846593 21406 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:13.877732 21259 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:13.878233 21338 raft_consensus.cc:2410] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251215 08:59:13.877111 23105 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
W20251215 08:59:13.878798 21406 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1289) took 3084 ms (client timeout 1742 ms). Trace:
W20251215 08:59:13.879242 21406 rpcz_store.cc:269] 1215 08:59:10.794583 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.806603 (+ 12020us) service_pool.cc:225] Handling call
1215 08:59:13.878575 (+3071972us) spinlock_profiling.cc:243] Waited 3.02 s on lock 0x7b54002f53d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:13.878691 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6950607232}
I20251215 08:59:13.846814 22872 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:13.881523 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:13.881898 22943 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.882449 22831 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:13.848413 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:13.851017 23123 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.852847 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:13.854812 21946 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:13.856112 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1478) took 1835 ms. Trace:
W20251215 08:59:13.857717 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 63194.6us
I20251215 08:59:13.858769 23014 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.862270 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:13.868851 23122 raft_consensus.cc:3055] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:13.879217 23105 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:13.882405 22943 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:13.954594 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.021997 22742 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:14.022552 23055 consensus_queue.cc:1059] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:14.023453 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.023648 22742 raft_consensus.cc:3055] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:14.023805 21336 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:14.024173 22742 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:14.024497 21336 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:14.024951 21336 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:14.025481 22742 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.027017 22742 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.024183 21341 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:13.977682 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.030390 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.030597 23122 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:14.032145 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.031908 23122 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.032758 21335 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:14.033486 23122 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.082481 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.084333 21407 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:14.059556 22977 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:14.085227 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:14.085266 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.078213 23048 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.086050 21411 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:14.077860 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.073791 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.067750 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:14.087308 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.086683 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:14.035662 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 92085.1us
I20251215 08:59:13.919836 21263 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:13.919606 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.045598 21336 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:14.097281 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:13.939348 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:13.892604 21409 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:14.033957 21259 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:14.093747 21412 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:14.101226 21332 raft_consensus.cc:1240] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:14.100808 21381 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.086381 21415 rpcz_store.cc:276] 1215 08:59:12.021057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:12.310866 (+289809us) service_pool.cc:225] Handling call
1215 08:59:12.325659 (+ 14793us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:12.326098 (+ 439us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:12.326132 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:13.685461 (+1359329us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b2c00092518. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:13.686024 (+ 563us) log.cc:844] Serialized 26 byte log entry
1215 08:59:13.750106 (+ 64082us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:59:13.750149 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:13.750252 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:13.855729 (+105477us) raft_consensus.cc:1685] finished
1215 08:59:13.855751 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:13.855951 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3657,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":6362,"spinlock_wait_cycles":8368512,"thread_start_us":62990,"threads_started":1,"wal-append.queue_time_us":167916}
I20251215 08:59:14.015350 23086 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:14.104106 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 249447us
I20251215 08:59:14.105901 21946 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:13.899178 23129 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:13.939078 22730 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.109308 21336 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:13.923935 23027 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
W20251215 08:59:14.110275 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:14.111136 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.085686 22977 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.113047 21407 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:14.110661 21338 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:14.054919 22831 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:13.882058 22872 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.120077 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.102680 23055 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:14.109942 23008 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:14.123508 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:14.125105 21335 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.119621 21229 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.128037 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.128334 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:14.128590 21265 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.110018 22950 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:14.129122 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.130344 23004 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.135777 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.130878 22976 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.131378 21406 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:14.124848 23008 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:14.147975 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.119930 23048 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.154361 21979 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:14.149169 21337 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:59:14.119182 23123 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.113082 23129 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.149837 22730 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.144906 22938 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.158257 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.159420 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.160246 23048 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.160614 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.137805 23131 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.161316 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.162593 22977 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.116073 23086 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.171643 21263 raft_consensus.cc:1275] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251215 08:59:14.131757 23055 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.137302 23027 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.181471 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.174666 21976 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:14.156499 22961 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:14.183318 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.228425 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.228644 21972 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:14.183832 22799 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.229966 22799 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.193692 21335 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:14.193151 23131 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.172734 22730 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.156272 23008 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.174062 23105 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.199266 23129 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.199936 22943 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.206157 22872 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.225863 21265 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
W20251215 08:59:14.253713 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 134131us
I20251215 08:59:14.233114 23014 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.176294 22950 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:14.259944 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:14.260802 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.264364 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.269536 21977 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:14.270071 21258 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:14.267062 21229 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.270498 21381 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.272737 21381 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:14.274601 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.278684 21265 raft_consensus.cc:2368] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3.
I20251215 08:59:14.272501 22872 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.285439 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.285902 22730 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.286623 22853 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:14.288051 23055 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.288473 21333 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:14.289244 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:14.289662 21333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1369) took 2465 ms (client timeout 1520 ms). Trace:
I20251215 08:59:14.289860 21406 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
W20251215 08:59:14.290035 21333 rpcz_store.cc:269] 1215 08:59:11.824693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:11.825089 (+ 396us) service_pool.cc:225] Handling call
1215 08:59:14.289481 (+2464392us) spinlock_profiling.cc:243] Waited 2.37 s on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:14.289582 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5448412416}
I20251215 08:59:14.277323 22951 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:14.258729 22961 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.309243 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.309559 22951 raft_consensus.cc:697] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:14.310635 22961 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.277745 22831 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.310784 22951 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.311573 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.300984 23123 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.309262 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.313654 21971 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:14.313805 22950 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.314597 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.315186 21977 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:14.315541 23086 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.302142 22674 consensus_queue.cc:1048] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:14.297739 21973 raft_consensus.cc:2410] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20251215 08:59:14.302459 21381 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:14.303630 23139 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20251215 08:59:14.306560 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.306946 23131 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.307579 22853 raft_consensus.cc:697] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:14.308030 22799 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.308406 23122 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.353879 21406 raft_consensus.cc:2410] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:14.353873 23122 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.351437 22961 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.355080 22961 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.355458 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.356307 21408 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:14.356966 22961 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.358011 22853 consensus_queue.cc:237] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.340904 21411 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:14.359752 22925 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.360252 22925 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.363018 22925 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.307605 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1128) took 3582 ms. Trace:
I20251215 08:59:14.363763 21340 rpcz_store.cc:276] 1215 08:59:10.725525 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.739299 (+ 13774us) service_pool.cc:225] Handling call
1215 08:59:14.292675 (+3553376us) spinlock_profiling.cc:243] Waited 3.55 s on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398cb6d58e 00007f398610c437 00007f398610c3c2 00007f398610c2f1 00007f398610c26c 00007f398610b522 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:14.307388 (+ 14713us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8166914816}
I20251215 08:59:14.338295 23123 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.342801 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.342680 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.343786 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.343950 23008 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.344678 23027 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.345136 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:14.345422 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 72451.8us
I20251215 08:59:14.370575 21379 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.372453 22799 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.374399 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.377162 22799 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:14.377635 22799 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:14.378094 22799 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.337230 23055 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.337914 22950 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.383067 23055 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.383172 22950 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.386107 22950 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.389617 21978 raft_consensus.cc:1240] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:14.394645 23008 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.396209 21258 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:14.382097 21379 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.348898 21946 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.350703 23150 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
W20251215 08:59:14.347812 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 45425.1us
W20251215 08:59:14.401206 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 52413.4us
I20251215 08:59:14.340814 21972 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251215 08:59:14.368243 23123 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.345602 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.375829 22674 consensus_queue.cc:1059] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:14.355753 23122 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:14.395712 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 58583.6us
I20251215 08:59:14.410210 23150 raft_consensus.cc:697] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:14.410480 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.411202 23150 consensus_queue.cc:237] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.411770 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.412380 21261 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:14.411640 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.415398 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.418747 21304 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.420836 21229 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.421692 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.422240 21259 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.422477 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.423247 21257 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:14.423360 21304 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.424099 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.424193 21304 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.422521 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.425364 21411 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:14.425488 21264 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:14.425678 23154 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:14.426054 23154 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.422255 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.426421 23154 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:14.422696 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.428285 21229 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.428462 21263 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.409080 23008 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:14.432931 23027 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.433655 23086 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:14.434777 23149 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:14.435070 23123 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.435874 22961 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:14.452347 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.438536 21406 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:14.427592 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.439582 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.439810 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.440210 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.461347 21413 raft_consensus.cc:2368] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4.
I20251215 08:59:14.449755 23027 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.415895 22951 raft_consensus.cc:2749] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:14.448119 21259 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:14.455757 21307 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.454303 22961 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.420635 23055 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.453002 23149 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:14.443465 23008 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.450323 23086 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.450501 22742 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:14.470861 21263 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
W20251215 08:59:14.471004 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 45780.9us
I20251215 08:59:14.471192 21381 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:14.452709 23131 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:14.482244 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.483269 21946 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.480454 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.474125 23008 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:14.485738 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.487056 23004 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:14.486982 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.469949 23129 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:14.492507 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 62691.3us
I20251215 08:59:14.468564 22961 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.461757 23123 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
W20251215 08:59:14.504335 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.472838 23122 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.472100 23149 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.509838 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.474325 22938 raft_consensus.cc:3055] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:14.480341 23086 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.519992 23086 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.520728 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.521247 22938 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:14.520987 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.525365 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.525679 22938 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.526029 21979 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:14.497005 23131 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.496847 23129 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:14.527155 23129 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:14.527169 21978 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:14.528153 23131 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.528371 23086 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.527081 22938 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.481205 22742 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.531939 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.474233 22895 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20251215 08:59:14.548631 23145 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:14.549120 23145 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.549468 23145 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.550416 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:14.552068 22951 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:14.552441 22951 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.552758 22951 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:14.557399 22799 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.558584 21381 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:14.559266 22799 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.559402 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:14.545590 21304 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:14.544003 23159 consensus_queue.cc:1059] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20251215 08:59:14.562242 23159 raft_consensus.cc:3055] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:14.562551 23159 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:14.563369 23159 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.564539 23159 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.531839 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:14.565934 21973 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:14.568758 21307 leader_election.cc:400] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3.
I20251215 08:59:14.569104 21307 leader_election.cc:403] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:14.569857 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.565282 22925 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:14.575940 23055 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 3
I20251215 08:59:14.576673 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.580065 22950 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:14.580476 23123 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:14.544710 22742 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.582221 21378 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.594595 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.605698 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.561198 23154 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.607216 23031 ts_tablet_manager.cc:1929] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251215 08:59:14.607831 23031 log.cc:1199] T 4f1c639a483f4827997eae6f3a6e375d P 0a2fcb89589c401d8efc8d86da4ba0f4: Deleting WAL directory at /tmp/dist-test-taskFXRoMa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765789040996898-20109-0/minicluster-data/ts-1-root/wals/4f1c639a483f4827997eae6f3a6e375d
I20251215 08:59:14.608673 23154 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.609974 23031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49730 (request call id 1126) took 3945 ms. Trace:
I20251215 08:59:14.610509 23031 rpcz_store.cc:276] 1215 08:59:10.674745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.715449 (+ 40704us) service_pool.cc:225] Handling call
1215 08:59:10.936895 (+221446us) ts_tablet_manager.cc:1151] Deleting tablet 4f1c639a483f4827997eae6f3a6e375d
1215 08:59:10.973224 (+ 36329us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1215 08:59:14.392530 (+3419306us) tablet_metadata.cc:641] Metadata flushed
1215 08:59:14.606974 (+214444us) tablet_metadata.cc:641] Metadata flushed
1215 08:59:14.609898 (+ 2924us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2710416,"tablet-delete.queue_time_us":180936,"thread_start_us":53403,"threads_started":1}
I20251215 08:59:14.613255 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.613514 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.613862 21335 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:14.593155 23123 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.629266 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 4f1c639a483f4827997eae6f3a6e375d (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:59:14.625485 22799 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:14.633563 22799 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:14.633126 23149 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.634068 22799 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.636193 21340 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:14.621631 22925 raft_consensus.cc:697] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:14.638861 22925 consensus_queue.cc:237] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.641656 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:14.645840 21332 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:14.647984 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.648268 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.649930 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.652086 23145 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.654800 23145 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.656129 21339 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:14.657850 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:14.626329 23123 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.659654 21333 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251215 08:59:14.597715 23027 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:14.673318 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 67714.8us
I20251215 08:59:14.673586 22938 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.674804 23144 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:14.675549 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.679878 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.680660 21257 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.683235 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.685070 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.686044 21339 raft_consensus.cc:2393] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:14.687049 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.688011 21333 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:14.690519 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:14.637673 23149 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.693076 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:14.694586 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.696583 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.697567 21339 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:14.698837 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.698913 23008 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.706188 22730 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:14.709789 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.701429 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.699735 21333 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:14.716784 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.717228 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.717375 21335 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:14.718371 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.724932 21333 raft_consensus.cc:2410] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:14.726656 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.730484 21337 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:14.644490 22961 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.729384 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:14.734112 22961 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.734030 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.735451 21339 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:14.733647 22799 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.739372 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.742664 21973 raft_consensus.cc:2393] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:14.751546 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:14.743177 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.743844 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.753854 21335 raft_consensus.cc:2393] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:14.755715 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.757647 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.758303 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:14.757189 23150 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:14.761924 22951 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:14.745071 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.745836 22799 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.791150 21333 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:14.747193 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.747342 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.748135 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.743983 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:14.884163 21334 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:14.771498 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.884690 21265 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:14.774370 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:14.775561 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.777529 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.780798 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.782382 23055 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3.
I20251215 08:59:14.783046 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.784524 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.784861 23008 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:14.909444 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.746510 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.723635 21379 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.846987 23027 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:14.869462 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:14.835161 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:14.926415 21409 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
W20251215 08:59:14.926658 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 203098us
I20251215 08:59:14.802883 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.793411 21259 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:14.845721 22742 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.769897 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:14.886901 21230 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.790005 23129 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.912854 21410 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:14.913465 21406 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:14.919859 22895 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:14.922559 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:14.911140 21944 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.166s (SENT) [suppressed 13 similar messages]
I20251215 08:59:14.910715 21336 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:14.912212 23163 log.cc:927] Time spent T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20251215 08:59:14.947860 21258 raft_consensus.cc:1275] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:14.948107 23129 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:14.948128 23159 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:14.948712 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 61901us
I20251215 08:59:14.949400 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.950117 21230 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:14.928736 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.938359 21332 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:14.940227 21264 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:14.941529 22742 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:14.894622 21411 raft_consensus.cc:3055] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:14.948789 21975 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:14.894754 21976 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 5
I20251215 08:59:14.926162 21266 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:14.894250 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:14.916735 21262 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:14.960309 21379 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:14.935582 22674 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:14.965114 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:14.974805 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:14.980139 21266 raft_consensus.cc:1240] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:14.976907 21265 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:14.956101 23004 raft_consensus.cc:697] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:14.956468 22853 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:14.958389 21411 raft_consensus.cc:740] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:15.004029 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:15.004837 21261 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:15.004978 21307 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:14.975435 22961 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:14.939235 23027 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:14.973896 23159 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:15.008023 22799 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:14.981069 22872 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251215 08:59:15.013214 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 52972.6us
I20251215 08:59:15.021522 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.015589 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:14.973598 22895 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.002004 22853 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:15.031914 21257 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:15.053295 22895 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.008868 22951 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:15.028620 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.059322 21307 leader_election.cc:400] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4.
I20251215 08:59:15.059597 21307 leader_election.cc:403] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:15.060463 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.006928 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:15.060850 22799 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:15.061314 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 45815.8us
I20251215 08:59:15.045598 21411 consensus_queue.cc:260] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.062112 21230 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.062259 23123 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.062817 21411 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:15.064234 23123 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:15.014187 23159 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.065239 23159 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:15.067458 23048 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.068636 21341 raft_consensus.cc:2368] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5.
I20251215 08:59:15.018935 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:15.068914 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 184429us
I20251215 08:59:14.977730 23144 consensus_queue.cc:1048] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20251215 08:59:15.030189 23055 consensus_queue.cc:1059] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.018s
I20251215 08:59:15.049211 23027 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.019240 23149 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20251215 08:59:15.012055 22742 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.046846 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.069355 21412 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:15.075266 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 53834.5us
I20251215 08:59:15.075860 21261 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:15.079970 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.033216 22853 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:15.030963 22961 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.008744 23004 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:15.093282 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:15.088467 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 82106.8us
I20251215 08:59:15.076500 23086 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:15.076298 22950 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.076427 23157 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:15.098610 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.101934 23123 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:15.083148 22925 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:15.114331 21260 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:15.118530 21381 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.120507 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.089399 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:15.126842 21340 raft_consensus.cc:1275] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251215 08:59:15.128258 21340 pending_rounds.cc:85] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 1
I20251215 08:59:15.128566 21340 pending_rounds.cc:107] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251215 08:59:15.128834 21340 raft_consensus.cc:2889] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:15.129580 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.177677 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:15.179773 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 117751us
W20251215 08:59:15.181352 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 101484us
I20251215 08:59:15.099671 23122 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20251215 08:59:15.183856 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.193877 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.195144 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:15.187983 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 58518.6us
I20251215 08:59:15.197942 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.203893 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:15.204640 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.206207 21232 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.210395 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.210273 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.140401 23173 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:15.211694 21409 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:15.182677 23004 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:15.212885 21337 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:15.213615 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.213951 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.203758 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.219297 21414 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:15.216939 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:15.181667 23148 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:15.220654 23148 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.222450 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:15.224278 22872 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:15.133203 22674 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.072s
I20251215 08:59:15.219339 21338 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:15.228468 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.181372 23098 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:15.200634 22925 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:15.235168 21337 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:15.238842 21944 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.224841 22872 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.232050 23157 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:15.242882 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.200716 23179 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.244294 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.246551 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.247617 21944 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.253518 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.227588 23123 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:15.222532 23148 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:15.229835 23055 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:15.233336 22951 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:15.260959 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 74671.8us
I20251215 08:59:15.270381 21260 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:15.240689 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.284531 21230 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.286835 21230 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.249653 23168 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:15.283159 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.299957 22925 raft_consensus.cc:697] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:15.300969 22925 consensus_queue.cc:237] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.241572 23098 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.261351 23182 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:15.312656 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 106557us
I20251215 08:59:15.312901 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.322466 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.289425 23027 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.336881 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:15.337564 21407 raft_consensus.cc:2393] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:15.305999 23179 raft_consensus.cc:2764] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251215 08:59:15.292188 23129 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:15.297824 23149 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:15.297880 23060 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.347668 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.353377 21413 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:15.308728 23122 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:15.290242 22951 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.361905 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.362248 21379 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.333349 23182 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.292706 23180 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20251215 08:59:15.332904 23181 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:15.363428 21946 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:15.355700 21410 raft_consensus.cc:2393] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:15.351173 23149 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.348601 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:15.395478 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 47888.3us
I20251215 08:59:15.396219 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.394410 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:15.409845 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 47740us
I20251215 08:59:15.411108 21413 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:15.415369 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:15.416184 21407 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.422343 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.428550 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.367483 23182 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:15.412967 23171 log.cc:927] Time spent T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20251215 08:59:15.397711 23149 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.355209 23129 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.442811 23154 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:15.394990 23086 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:15.449750 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:15.417012 23174 log.cc:927] Time spent T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
W20251215 08:59:15.448843 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 51657.3us
I20251215 08:59:15.394048 23008 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:15.448791 23055 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.412181 23105 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:15.368036 23122 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:15.391458 23181 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:15.409631 23173 raft_consensus.cc:697] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:15.414000 23123 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.454926 23194 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.469939 23123 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.472342 23123 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:15.473688 23173 consensus_queue.cc:237] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.474726 21976 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:15.459892 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.443598 23185 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:15.475519 23004 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.464198 23086 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.480268 23178 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:15.480619 23122 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:15.446359 23184 log.cc:927] Time spent T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
I20251215 08:59:15.469182 22927 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.136s
I20251215 08:59:15.487608 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.489286 23193 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:15.489342 23195 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:15.489770 23193 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.490016 23195 raft_consensus.cc:697] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:15.490159 23193 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:15.491233 23159 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.491016 23195 consensus_queue.cc:237] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.492419 21335 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.493569 23197 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:59:15.494696 22872 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.499197 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.501911 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.503749 23123 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.502218 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.507877 21335 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:15.473709 21379 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.511814 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.514009 23178 raft_consensus.cc:697] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:15.514978 23178 consensus_queue.cc:237] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.514106 22951 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:15.464581 23008 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.506928 21341 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
W20251215 08:59:15.523747 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 50127.5us
I20251215 08:59:15.442162 23148 consensus_queue.cc:1048] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251215 08:59:15.529660 21407 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:15.534024 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.535708 21412 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:15.497941 22950 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4.
I20251215 08:59:15.535959 21333 raft_consensus.cc:2393] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:15.534874 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.537624 21337 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:15.538949 21307 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:15.540331 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.515002 23098 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:15.478649 23181 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.505425 22853 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.547678 21944 leader_election.cc:400] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5.
I20251215 08:59:15.480857 23086 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.543563 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.468595 23055 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.528887 23008 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.534042 23105 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:15.557848 21944 leader_election.cc:403] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:15.466852 23183 consensus_queue.cc:1059] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20251215 08:59:15.543177 21379 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.544754 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.543529 23123 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:15.560739 21414 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:15.554724 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:15.561044 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.570188 21407 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:15.544965 21307 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:15.573516 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 262725us
W20251215 08:59:15.576385 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 288347us
I20251215 08:59:15.576936 21230 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.582319 23202 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:15.553736 21411 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:15.557269 23105 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.583503 23048 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:15.586546 23182 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.592391 23027 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:15.593393 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 256632us
I20251215 08:59:15.594182 22853 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:15.597208 22872 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:15.597652 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:15.609148 21230 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.622077 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:15.600785 23122 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:15.631352 21230 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.623298 23105 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.621245 23123 raft_consensus.cc:697] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:15.623978 23048 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:15.649595 23048 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:15.649852 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:15.655386 23207 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.658169 23157 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.658612 23157 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.658715 23182 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:15.633749 23129 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:15.632320 23187 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:15.661897 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.632486 21230 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.624958 21232 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.639297 23159 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.624554 23195 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.645478 22742 raft_consensus.cc:3055] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:15.645855 23149 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:15.672890 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 48052.5us
I20251215 08:59:15.673115 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.673822 21333 raft_consensus.cc:2368] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:15.674407 23149 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.667311 22872 raft_consensus.cc:2764] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251215 08:59:15.675014 21232 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.623586 23202 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:15.647164 23201 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:15.648162 23181 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.637832 23122 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:15.666832 21332 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
W20251215 08:59:15.667265 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 119769us
I20251215 08:59:15.647199 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:15.672742 22742 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:15.675151 23195 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.669628 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:15.686949 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 111831us
I20251215 08:59:15.688503 22872 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:15.689441 23149 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:15.713755 23055 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.691306 21230 leader_election.cc:400] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:15.714767 21230 leader_election.cc:403] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251215 08:59:15.691745 23179 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.716334 23179 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.717494 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.693506 21339 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251215 08:59:15.719310 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.719816 21334 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:15.693363 21232 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
W20251215 08:59:15.692890 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 303501us
I20251215 08:59:15.648941 23210 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:15.705811 23008 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.708104 23202 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:15.665318 23182 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.691942 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.727607 21333 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.721271 21304 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.721464 21232 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.677749 23201 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:15.722533 21229 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:15.724814 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:15.726017 21230 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.731143 21232 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.731776 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.732089 23209 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:15.732445 21261 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:15.732649 23209 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:15.733605 23209 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.729290 23173 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:15.735066 22961 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:15.732281 23210 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:15.738189 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 52159.2us
I20251215 08:59:15.723820 22872 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.739321 21944 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.722136 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:15.726574 23149 raft_consensus.cc:2764] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20251215 08:59:15.735575 23086 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:15.748452 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.744843 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.732110 23178 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:15.737447 22961 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.735615 23201 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.758263 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:15.742769 23199 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.787443 21978 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:15.773025 23217 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.799515 21333 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:15.802712 21230 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.734989 23182 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.727741 23191 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20251215 08:59:15.805735 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 74683.3us
I20251215 08:59:15.752537 23129 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:15.753204 23086 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:15.757772 21410 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.735293 23173 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.812486 21232 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:15.756347 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.778030 22961 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.816388 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.816597 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:15.735332 23122 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.822554 23207 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5.
I20251215 08:59:15.824268 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:15.824874 21978 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.825860 23221 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:15.826324 21946 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:15.810741 21976 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:15.806867 23178 raft_consensus.cc:697] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:15.811146 23086 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:15.851574 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.829574 21333 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:15.828701 21946 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:15.838743 21261 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:15.812150 23183 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.854493 21230 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:15.855024 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:15.841404 23199 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:15.843163 21978 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:15.845296 23221 raft_consensus.cc:697] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:15.848234 23205 log.cc:927] Time spent T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20251215 08:59:15.848387 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:15.849836 23055 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:15.850603 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.851246 23182 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.838953 23008 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.848268 21262 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:15.875974 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:15.901175 21263 raft_consensus.cc:2368] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4.
I20251215 08:59:15.875964 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:15.878829 21304 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.758489 23123 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.812369 23149 consensus_queue.cc:1059] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20251215 08:59:15.886515 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:15.897167 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 141120us
I20251215 08:59:15.799531 22742 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.879870 22895 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:15.879359 23199 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:15.891767 23225 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:15.911608 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:15.913139 21333 raft_consensus.cc:1275] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:15.914288 21340 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.884725 23181 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.817979 23193 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.916810 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.915602 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:15.915457 23098 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:15.920243 23179 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.920506 21263 raft_consensus.cc:1275] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:15.920514 23098 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:15.910084 23159 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:15.893088 23207 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20251215 08:59:15.891904 23203 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:15.890434 22961 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:15.912853 23211 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:15.937867 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 193147us
I20251215 08:59:15.917297 23182 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.939446 21335 raft_consensus.cc:2368] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:15.913527 23199 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:15.942965 23149 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:15.943164 21975 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:15.913919 23225 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:15.909349 23055 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.945358 22938 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20251215 08:59:15.957000 21258 raft_consensus.cc:2368] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:15.949828 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.949947 23098 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:15.950428 23221 consensus_queue.cc:237] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.904968 23183 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.952973 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:15.961445 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:15.963433 21341 raft_consensus.cc:2393] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:15.965229 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:15.945874 23182 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:15.924552 23210 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20251215 08:59:15.940868 23195 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:15.984529 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:15.986846 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 108097us
I20251215 08:59:15.955449 23225 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:15.940816 23008 consensus_queue.cc:1048] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20251215 08:59:15.986018 22895 raft_consensus.cc:697] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:15.952378 23149 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:16.005622 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 101520us
I20251215 08:59:16.000986 22927 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20251215 08:59:16.009073 22895 consensus_queue.cc:237] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.010329 21340 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
W20251215 08:59:15.997431 23216 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20251215 08:59:16.011721 21944 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:15.996699 23173 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:15.941272 23178 consensus_queue.cc:237] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.022106 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:16.022645 21340 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:16.023548 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:16.019048 23228 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:16.015923 23209 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:16.079210 22872 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:16.079766 23105 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.081887 23105 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:16.082365 23210 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:16.082742 23210 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.083073 23210 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:16.077340 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 53874.9us
I20251215 08:59:16.095095 23211 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.095809 23211 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.097777 22961 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:16.097833 23209 raft_consensus.cc:697] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:16.099354 23209 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.110944 21340 raft_consensus.cc:1275] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:16.116338 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:16.132092 21261 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:16.133926 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:16.146857 21333 raft_consensus.cc:2393] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:16.135322 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:16.139294 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:16.147742 21407 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:16.140228 21412 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251215 08:59:16.128242 23195 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:16.150669 21340 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 5
I20251215 08:59:16.171166 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:16.171602 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:16.171880 21413 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:16.203445 21407 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
W20251215 08:59:16.220875 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 65098.7us
I20251215 08:59:16.140862 23055 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.172449 23195 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.219288 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:16.302011 21408 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:16.175573 23202 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:16.172541 23149 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.217562 21410 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
W20251215 08:59:16.309597 23157 consensus_meta.cc:287] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.651s user 0.006s sys 0.007s
W20251215 08:59:16.462498 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 465920us
I20251215 08:59:16.481508 21334 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:16.350209 23055 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:16.513847 21334 raft_consensus.cc:1275] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:16.448251 23149 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.493330 23211 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.518918 23159 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:16.481087 23157 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.602092 23055 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.697440 23237 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:16.697736 23221 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.000s sys 0.005s
I20251215 08:59:16.358408 23199 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.696158 21414 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:16.726660 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:16.668213 23200 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:16.687717 21972 consensus_meta.cc:287] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.830s user 0.006s sys 0.006s
I20251215 08:59:16.742537 22950 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:16.687968 23048 consensus_meta.cc:287] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.038s user 0.008s sys 0.005s
I20251215 08:59:16.751029 21972 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:16.774588 23048 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.777328 21260 raft_consensus.cc:1240] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:16.634953 23149 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.745325 21978 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:16.700225 23201 consensus_meta.cc:287] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.890s user 0.002s sys 0.010s
I20251215 08:59:16.779084 23201 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.779620 23149 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:16.780853 23201 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.746376 21337 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251215 08:59:16.749862 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:16.695582 23086 consensus_meta.cc:287] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.844s user 0.004s sys 0.006s
I20251215 08:59:16.788125 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:16.809319 23048 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:16.811604 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:16.827804 21262 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
W20251215 08:59:16.780903 23210 consensus_meta.cc:287] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.697s user 0.006s sys 0.002s
I20251215 08:59:16.852216 23185 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.852303 23251 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20251215 08:59:16.853077 23185 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.782826 23200 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:16.855897 23185 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:16.856284 23157 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.856904 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1144) took 1597 ms. Trace:
I20251215 08:59:16.857396 21338 rpcz_store.cc:276] 1215 08:59:15.259687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.259816 (+ 129us) service_pool.cc:225] Handling call
1215 08:59:16.856341 (+1596525us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398cb6d58e 00007f398610c437 00007f398610c3c2 00007f398610c2f1 00007f398610c26c 00007f398610b522 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:16.856817 (+ 476us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":776788736}
I20251215 08:59:16.858523 23200 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:16.743875 23221 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s user 0.000s sys 0.006s
W20251215 08:59:16.867439 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 172093us
I20251215 08:59:16.408823 21340 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:16.860450 23178 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.006s sys 0.003s
I20251215 08:59:16.743302 23055 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:16.685494 23159 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:16.876411 21265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.025s sys 0.028s
W20251215 08:59:16.876857 21265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.866s user 0.025s sys 0.028s
I20251215 08:59:16.861778 23195 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.836354 22950 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.838738 23086 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:16.873158 23178 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.006s sys 0.003s
I20251215 08:59:16.859222 23210 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:16.928475 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 231847us
I20251215 08:59:16.919302 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:16.928979 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:16.928151 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 464699us
I20251215 08:59:16.933511 21264 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:16.919714 21259 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:16.938711 21338 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:16.905952 21408 consensus_meta.cc:287] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.603s user 0.009s sys 0.002s
I20251215 08:59:16.941430 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:16.746340 23237 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:16.943851 21333 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251215 08:59:16.944365 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:16.944495 23180 raft_consensus.cc:3055] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:16.945039 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:16.947031 23233 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.947587 23250 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.947899 23230 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:16.953949 23055 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.990479 21230 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:16.954051 23187 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:16.955668 23185 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.994513 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1331) took 1140 ms. Trace:
I20251215 08:59:16.994879 21262 rpcz_store.cc:276] 1215 08:59:15.853645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.906337 (+ 52692us) service_pool.cc:225] Handling call
1215 08:59:15.913058 (+ 6721us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:16.828378 (+915320us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:16.994273 (+165895us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b5400323f54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:16.994451 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2103132032}
I20251215 08:59:16.956405 23199 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.959332 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:16.960893 22950 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.963068 23086 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:16.969245 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1506) took 1750 ms. Trace:
I20251215 08:59:17.001137 21409 rpcz_store.cc:276] 1215 08:59:15.218689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.252380 (+ 33691us) service_pool.cc:225] Handling call
1215 08:59:15.275578 (+ 23198us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:15.349038 (+ 73460us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:15.349070 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:15.349113 (+ 43us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:15.349148 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:16.968941 (+1619793us) spinlock_profiling.cc:243] Waited 73.2 ms on lock 0x7b5400341a54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:16.969004 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:16.969176 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":168435072}
I20251215 08:59:16.981706 21266 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:16.983170 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 49186.9us
I20251215 08:59:16.965787 21408 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:16.994174 21230 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:16.957197 23157 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:16.997743 21259 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:16.970415 23180 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:16.971241 23182 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:16.981841 23178 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.011595 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:16.983690 22895 consensus_queue.cc:1048] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20251215 08:59:17.017812 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:17.021243 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 153114us
I20251215 08:59:17.022079 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:17.022511 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:17.023489 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.023979 21409 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:17.024060 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.009945 23251 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:17.030398 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:17.031127 21407 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:17.036119 21334 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:17.040429 21414 raft_consensus.cc:1275] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:17.004160 23055 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.049767 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:16.991354 23250 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.053777 21379 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:17.011623 23185 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.012382 23157 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.033304 23086 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:17.058533 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.061693 23261 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.039098 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:17.053973 23195 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:17.062215 23261 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.062809 23195 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:17.063165 23195 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:17.064324 23261 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:17.064846 23241 log.cc:927] Time spent T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.279s user 0.001s sys 0.001s
W20251215 08:59:17.065773 23225 consensus_meta.cc:287] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 1.068s user 0.016s sys 0.000s
I20251215 08:59:17.065896 23257 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.066409 23225 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.067451 23224 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:17.067907 23224 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.068310 23224 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:17.029362 23209 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.069341 23209 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.070741 23244 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:17.096864 23244 raft_consensus.cc:697] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:17.013849 23178 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.102684 23262 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:17.071846 23209 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.071967 23237 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.072625 23240 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:17.103766 23178 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:17.066529 23257 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.052711 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:17.106665 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:17.114075 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 61448.4us
I20251215 08:59:17.061915 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:16.992082 23233 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.077456 23222 consensus_queue.cc:1048] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20251215 08:59:17.078289 23246 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
W20251215 08:59:17.079986 23242 log.cc:927] Time spent T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20251215 08:59:17.000627 23187 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.005939 23230 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.082438 23055 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.074635 23159 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:17.084631 23238 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20251215 08:59:17.085292 23210 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.087023 23250 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.088620 23086 raft_consensus.cc:697] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:17.090821 21406 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1509) took 1792 ms (client timeout 2260 ms). Trace:
I20251215 08:59:17.094081 22742 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:17.094237 23185 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.095383 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1350) took 1452 ms. Trace:
I20251215 08:59:15.984225 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251215 08:59:17.099557 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 45856.4us
I20251215 08:59:17.118702 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:17.186870 23230 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:17.176146 21324 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f1c639a483f4827997eae6f3a6e375d with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:49730
I20251215 08:59:17.177246 23193 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:17.189522 21341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1540) took 1277 ms (client timeout 1639 ms). Trace:
W20251215 08:59:17.189879 21341 rpcz_store.cc:269] 1215 08:59:15.911800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.912003 (+ 203us) service_pool.cc:225] Handling call
1215 08:59:17.189331 (+1277328us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b54001252d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:17.189451 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99981056}
I20251215 08:59:17.189851 23086 consensus_queue.cc:237] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.193638 21261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1329) took 1340 ms. Trace:
I20251215 08:59:17.193990 21261 rpcz_store.cc:276] 1215 08:59:15.853123 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.853203 (+ 80us) service_pool.cc:225] Handling call
1215 08:59:17.193489 (+1340286us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b5400170054. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:17.193588 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":409519616}
I20251215 08:59:17.195319 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 139) took 1464 ms. Trace:
I20251215 08:59:17.202266 21978 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20251215 08:59:17.100356 21332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1423) took 1912 ms (client timeout 1701 ms). Trace:
I20251215 08:59:17.101125 23244 consensus_queue.cc:237] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:17.102738 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1532) took 1433 ms. Trace:
W20251215 08:59:17.175648 21258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1326) took 1325 ms (client timeout 1755 ms). Trace:
I20251215 08:59:17.018927 23231 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20251215 08:59:17.064652 23157 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.175834 23237 consensus_queue.cc:1048] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
W20251215 08:59:17.182734 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 67439.5us
W20251215 08:59:17.203183 21332 rpcz_store.cc:269] 1215 08:59:15.185156 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.185274 (+ 118us) service_pool.cc:225] Handling call
1215 08:59:17.097405 (+1912131us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400298454. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:17.097538 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":803081600}
I20251215 08:59:17.204303 21337 rpcz_store.cc:276] 1215 08:59:15.668830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.675799 (+ 6969us) service_pool.cc:225] Handling call
1215 08:59:15.676126 (+ 327us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:16.781772 (+1105646us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:17.102493 (+320721us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540009d3d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:17.102680 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2460640640}
I20251215 08:59:17.205631 21266 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
W20251215 08:59:17.206209 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 153014us
I20251215 08:59:17.195640 21260 rpcz_store.cc:276] 1215 08:59:15.731104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.731242 (+ 138us) service_pool.cc:225] Handling call
1215 08:59:15.731534 (+ 292us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:16.777797 (+1046263us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:17.195101 (+417304us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540020b054. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:17.195261 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2404513536}
I20251215 08:59:17.217903 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:17.218168 21946 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:17.088718 21162 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36408 (request call id 212) took 12339 ms (client timeout 15000 ms). Trace:
I20251215 08:59:17.176199 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 159) took 1428 ms. Trace:
W20251215 08:59:17.204367 21258 rpcz_store.cc:269] 1215 08:59:15.850272 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.850373 (+ 101us) service_pool.cc:225] Handling call
1215 08:59:17.175373 (+1325000us) spinlock_profiling.cc:243] Waited 71.2 ms on lock 0x7b5400356e54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:17.175581 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":163702144}
I20251215 08:59:17.219229 21335 rpcz_store.cc:276] 1215 08:59:15.747956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.748145 (+ 189us) service_pool.cc:225] Handling call
1215 08:59:17.176024 (+1427879us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b540009bfd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:17.176144 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":427681152}
W20251215 08:59:17.124010 23258 log.cc:927] Time spent T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20251215 08:59:17.120309 23179 raft_consensus.cc:3055] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:17.118980 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.231253 21259 raft_consensus.cc:2410] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20251215 08:59:17.255388 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:17.182941 21972 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46170 (request call id 348) took 2113 ms (client timeout 2052 ms). Trace:
I20251215 08:59:17.204056 23225 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:17.255401 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:17.263070 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.244186 21230 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:17.258507 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.259583 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:17.265409 21412 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:17.256083 21413 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:17.216535 23230 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:17.266927 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 81412.3us
I20251215 08:59:17.127192 23248 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:17.228667 23131 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:17.228446 23157 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.241154 22938 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:17.230252 23027 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:17.270548 21381 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:17.120558 23257 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:17.255123 23173 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:17.276544 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 46722.6us
I20251215 08:59:17.292783 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:17.309686 21413 raft_consensus.cc:2393] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:17.241215 23179 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:17.311730 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.313555 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.315183 21149 catalog_manager.cc:4985] TS 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): tablet 4f1c639a483f4827997eae6f3a6e375d (table test-workload [id=e4317e71ee4e47f08ded586850d0d416]) successfully deleted
I20251215 08:59:17.313162 23233 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.334280 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 474) took 1952 ms. Trace:
I20251215 08:59:17.335646 23268 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:17.331184 23267 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:17.336412 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 92314.8us
I20251215 08:59:17.312435 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:17.337263 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.337859 21414 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:17.336625 23267 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.338675 23190 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.338764 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.338021 22895 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:17.339537 21414 raft_consensus.cc:2393] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:17.319087 21334 raft_consensus.cc:1275] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:17.329048 21337 raft_consensus.cc:1275] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:17.342615 21412 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:17.341964 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:17.343518 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:17.260773 23209 consensus_queue.cc:1059] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20251215 08:59:17.346354 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:17.273115 23212 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.271950 23211 consensus_queue.cc:1059] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20251215 08:59:17.348383 21265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1507) took 2090 ms. Trace:
I20251215 08:59:17.304791 23157 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:17.242777 21406 rpcz_store.cc:269] 1215 08:59:15.298697 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.310860 (+ 12163us) service_pool.cc:225] Handling call
1215 08:59:17.090546 (+1779686us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54002f53d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:17.090737 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":683465088}
I20251215 08:59:17.401607 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.343361 21334 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:17.376784 21975 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
I20251215 08:59:17.379130 23008 raft_consensus.cc:3055] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:17.383281 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:17.218742 21162 rpcz_store.cc:269] 1215 08:59:04.748808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:04.768280 (+ 19472us) service_pool.cc:225] Handling call
1215 08:59:17.088639 (+12320359us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:14.232051 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:14.244408 (+ 12357us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:14.244475 (+ 67us) write_op.cc:435] Acquired schema lock
1215 08:59:14.244500 (+ 25us) tablet.cc:660] Decoding operations
1215 08:59:14.244968 (+ 468us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:14.245096 (+ 128us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:14.245127 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
1215 08:59:14.245454 (+ 327us) tablet.cc:699] Row locks acquired
1215 08:59:14.245476 (+ 22us) write_op.cc:260] PREPARE: finished
1215 08:59:14.245662 (+ 186us) write_op.cc:270] Start()
1215 08:59:14.245840 (+ 178us) write_op.cc:276] Timestamp: P: 1765789154245588 usec, L: 0
1215 08:59:14.245879 (+ 39us) op_driver.cc:348] REPLICATION: starting
1215 08:59:14.246702 (+ 823us) log.cc:844] Serialized 541 byte log entry
1215 08:59:14.583382 (+336680us) op_driver.cc:464] REPLICATION: finished
1215 08:59:14.983865 (+400483us) write_op.cc:301] APPLY: starting
1215 08:59:15.005569 (+ 21704us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:15.010675 (+ 5106us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:15.049196 (+ 38521us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:15.049514 (+ 318us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:15.049782 (+ 268us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1215 08:59:15.049848 (+ 66us) write_op.cc:312] APPLY: finished
1215 08:59:15.050098 (+ 250us) log.cc:844] Serialized 23 byte log entry
1215 08:59:15.050312 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:15.050421 (+ 109us) write_op.cc:454] Released schema lock
1215 08:59:15.050482 (+ 61us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:15.050911 (+ 429us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:15.050950 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1421621,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":371455,"apply.run_cpu_time_us":7253,"apply.run_wall_time_us":77049,"num_ops":1,"prepare.queue_time_us":321385,"prepare.run_cpu_time_us":48110,"prepare.run_wall_time_us":237607,"raft.queue_time_us":96140,"raft.run_cpu_time_us":111482,"raft.run_wall_time_us":258065,"replication_time_us":337349,"thread_start_us":669779,"threads_started":4,"wal-append.queue_time_us":219137,"wal-append.run_cpu_time_us":29174,"wal-append.run_wall_time_us":1640440}]]}
I20251215 08:59:17.406298 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:17.271072 23230 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.408511 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:17.332839 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.412189 21232 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:17.206986 21411 rpcz_store.cc:276] 1215 08:59:15.642578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.643108 (+ 530us) service_pool.cc:225] Handling call
1215 08:59:15.643423 (+ 315us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:15.662392 (+ 18969us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:15.909785 (+247393us) log.cc:844] Serialized 14 byte log entry
1215 08:59:15.921498 (+ 11713us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:15.921549 (+ 51us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:15.921593 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:17.095053 (+1173460us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b5400341054. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:17.095122 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:17.095298 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":72566,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":2587,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":42916096,"thread_start_us":241978,"threads_started":1}
I20251215 08:59:17.305830 22938 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.403090 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:17.329267 23189 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.330201 23027 raft_consensus.cc:697] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:17.335453 21980 rpcz_store.cc:276] 1215 08:59:15.274069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.274201 (+ 132us) service_pool.cc:225] Handling call
1215 08:59:15.300074 (+ 25873us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:15.300272 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 7
1215 08:59:15.344784 (+ 44512us) log.cc:844] Serialized 14 byte log entry
1215 08:59:15.345235 (+ 451us) log.cc:844] Serialized 14 byte log entry
1215 08:59:15.811825 (+466590us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b1800369880. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:17.078847 (+1267022us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b1800369880. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984a7b33a 00007f398e1005c9 00007f398776a110 00007f398775d633 00007f398775308a 00007f39877535da 00007f3987bbe6aa 00007f3987bcdd3f 00007f3987bcdcb8
1215 08:59:17.225837 (+146990us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:17.225888 (+ 51us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:17.225927 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:17.226172 (+ 245us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:17.226279 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9124,"prepare.queue_time_us":92516,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":5567,"raft.queue_time_us":2321,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":1845,"spinlock_wait_cycles":8589440,"thread_start_us":146668,"threads_started":1}
W20251215 08:59:17.404536 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 63103.2us
I20251215 08:59:17.407509 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:17.377049 23195 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.334870 23187 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:17.405527 21338 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:17.427138 21412 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:17.410003 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:17.413550 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 156797us
I20251215 08:59:17.380225 23233 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.416035 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:17.435518 23248 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.383128 23060 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:17.435940 21339 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:17.436151 23060 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.436585 22938 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:17.437233 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.438851 23220 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.439376 23220 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.439968 23183 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.442775 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.444307 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.445003 21339 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:17.418877 22961 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.378085 23190 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.419361 23267 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.333501 23224 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:17.258538 21972 rpcz_store.cc:269] 1215 08:59:15.069754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.070570 (+ 816us) service_pool.cc:225] Handling call
1215 08:59:15.647047 (+576477us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:17.182708 (+1535661us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54003f11d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:17.182836 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369276544}
I20251215 08:59:17.405287 23008 raft_consensus.cc:740] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:17.409312 23200 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:17.425827 21946 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:17.429219 21415 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:17.424698 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:17.429744 21307 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:17.413100 23201 consensus_queue.cc:1059] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.010s
I20251215 08:59:17.404932 23179 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.399667 21265 rpcz_store.cc:276] 1215 08:59:15.258240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.258947 (+ 707us) service_pool.cc:225] Handling call
1215 08:59:15.259233 (+ 286us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:15.342705 (+ 83472us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:15.342749 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:15.356815 (+ 14066us) log.cc:844] Serialized 26 byte log entry
1215 08:59:16.010406 (+653591us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b18002d67c0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:16.877515 (+867109us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:16.877574 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:16.877672 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:17.340170 (+462498us) raft_consensus.cc:1685] finished
1215 08:59:17.340190 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:17.348302 (+ 8112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80871,"raft.queue_time_us":131689,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":15984,"spinlock_wait_cycles":6670720,"thread_start_us":866881,"threads_started":1,"wal-append.queue_time_us":1559275}
I20251215 08:59:17.446408 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.417421 23048 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:17.421187 23129 raft_consensus.cc:3055] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:17.412175 23250 consensus_queue.cc:1059] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20251215 08:59:17.413111 23191 consensus_queue.cc:1059] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20251215 08:59:17.452265 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.453117 22961 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.453653 23173 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:17.453171 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:17.452291 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 66694.4us
I20251215 08:59:17.461650 21262 raft_consensus.cc:1275] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:17.471206 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:17.455147 23236 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.388212 23212 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:17.496784 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 67124.7us
I20251215 08:59:17.454540 23027 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.410778 22895 raft_consensus.cc:697] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:17.448415 23211 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:17.451076 23178 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:17.502080 21261 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
W20251215 08:59:17.495257 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 44302.3us
I20251215 08:59:17.522719 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.545544 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:17.525822 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.526240 21260 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:17.503618 21264 raft_consensus.cc:2410] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:17.534016 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.496145 23267 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:17.535805 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.497831 23202 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.436019 23248 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.540109 22742 consensus_queue.cc:1059] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:17.550829 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:17.447978 23233 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.546244 21262 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:17.546461 21412 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:17.499799 23183 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:17.503654 23129 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:17.549893 21263 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:17.502827 23004 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.533533 21973 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:17.540324 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:17.537319 23179 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:17.496656 23200 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.560813 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.538766 23231 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.499974 22853 consensus_queue.cc:1059] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20251215 08:59:17.537922 23251 consensus_queue.cc:1175] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f0824fa5450345719cac22fc6ae3ce2e" responder_term: 4 status { last_received { term: 4 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 4 index: 7 } }
I20251215 08:59:17.422884 23189 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.647027 21266 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:17.548396 23048 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.548807 21257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1526) took 1774 ms. Trace:
W20251215 08:59:17.625269 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 65223.5us
W20251215 08:59:17.674619 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 134365us
I20251215 08:59:17.586979 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.685642 21260 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:17.689795 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:17.595058 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:17.702101 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:17.661623 23275 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.231s user 0.001s sys 0.000s
I20251215 08:59:17.553164 23131 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:17.557228 23244 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.716459 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:17.721575 21978 raft_consensus.cc:2376] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5. Re-sending same reply.
I20251215 08:59:17.627061 23193 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.558722 21408 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 6. Ops: []
I20251215 08:59:17.666921 23212 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.722007 21266 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:17.723423 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.673882 23060 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.561267 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:17.715903 23183 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:17.688519 23048 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:17.680050 23159 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:17.684746 23267 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.735514 21379 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:17.686092 23157 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:17.558812 23202 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.542829 23236 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.555837 23008 consensus_queue.cc:260] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.719241 23220 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.751401 21264 raft_consensus.cc:1275] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:17.680779 23231 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.757591 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.729976 23237 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.681186 23004 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.766788 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:17.733927 23230 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:17.740633 23190 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.739333 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:17.773267 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.740475 23195 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.739996 23212 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:17.729382 23266 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20251215 08:59:17.773463 21259 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:17.751794 21406 raft_consensus.cc:606] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:17.729568 23257 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251215 08:59:17.816942 21262 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:17.816831 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.754618 22895 consensus_queue.cc:237] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.819880 21265 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:17.820499 23220 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.768046 23207 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.730679 23244 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.734189 23193 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:17.820062 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 106588us
I20251215 08:59:17.765607 23268 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:17.754793 22742 consensus_queue.cc:1059] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.067s
I20251215 08:59:17.773792 23267 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.755545 23191 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.810647 23199 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.754881 23224 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.819463 23283 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.835729 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:17.836413 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 97619.9us
I20251215 08:59:17.827175 22853 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:17.811108 23159 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20251215 08:59:17.752103 23129 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.821640 23237 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.829167 23008 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:17.853839 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.930754 21260 raft_consensus.cc:2393] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:17.831166 21406 raft_consensus.cc:614] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:17.831187 21972 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
W20251215 08:59:17.947594 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:17.933727 21307 leader_election.cc:400] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:17.688253 21257 rpcz_store.cc:276] 1215 08:59:15.774324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.775048 (+ 724us) service_pool.cc:225] Handling call
1215 08:59:15.775487 (+ 439us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:15.775746 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 2
1215 08:59:15.811142 (+ 35396us) log.cc:844] Serialized 14 byte log entry
1215 08:59:17.178029 (+1366887us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b18002fa3a0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:17.420634 (+242605us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b18002fa3a0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b03045 00007f3987babce0 00007f3987bbe396 00007f3987bcef07 00007f3987bcee88
1215 08:59:17.498909 (+ 78275us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:17.509830 (+ 10921us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:17.522647 (+ 12817us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:17.536054 (+ 13407us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:17.542738 (+ 6684us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84585,"prepare.queue_time_us":30093,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":2359,"raft.queue_time_us":9780,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"spinlock_wait_cycles":20485632,"thread_start_us":74995,"threads_started":1}
I20251215 08:59:17.832666 22961 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.831816 23195 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:17.912544 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 152273us
I20251215 08:59:17.962553 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:17.963246 21335 raft_consensus.cc:2376] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4. Re-sending same reply.
I20251215 08:59:17.829190 23284 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.953119 21307 leader_election.cc:403] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:17.968988 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.969244 21977 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:17.855082 23248 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.857788 23220 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.940546 23199 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.969601 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:17.857517 23291 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.840386 23207 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:17.945348 23282 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.222s user 0.000s sys 0.001s
I20251215 08:59:17.990304 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:17.834288 23212 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.856048 23200 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.933065 22853 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:17.936188 23268 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:17.878257 23157 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.977645 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:17.989646 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:17.996364 21259 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:17.848263 23004 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.854058 23202 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.855430 23193 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.976150 23220 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.991360 21980 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251215 08:59:17.999861 23195 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:17.990029 23230 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20251215 08:59:17.998171 23292 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:17.998437 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:17.879607 23287 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:17.936506 23233 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:18.008949 21944 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.006s
W20251215 08:59:18.058218 21944 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.000s sys 0.006s
I20251215 08:59:18.014583 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:18.015780 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 47806.8us
W20251215 08:59:18.058974 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 44503.5us
W20251215 08:59:17.976995 23281 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20251215 08:59:17.979380 23199 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.000041 23248 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:17.978850 23291 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.973845 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:18.037909 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 180572us
I20251215 08:59:17.984683 23244 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:17.953476 23191 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:17.983520 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:18.018858 22853 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:18.025357 23159 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20251215 08:59:18.045085 23294 raft_consensus.cc:3055] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:18.028437 23268 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:18.016846 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 83300.4us
I20251215 08:59:18.091802 21259 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:18.058008 21406 raft_consensus.cc:606] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:18.033913 23157 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.030290 23284 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:17.971930 23283 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:18.085650 23193 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:18.118070 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:18.102025 23199 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:18.114012 23294 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:18.265883 21341 raft_consensus.cc:2410] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251215 08:59:18.057304 23292 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:18.298261 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:18.167008 23193 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.256567 23027 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:18.264544 23244 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.264194 23199 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.060387 23287 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:18.424782 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 157012us
W20251215 08:59:18.649708 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:18.432629 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:18.552368 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:18.652568 21407 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1364) took 1607 ms (client timeout 1792 ms). Trace:
W20251215 08:59:18.653110 21407 rpcz_store.cc:269] 1215 08:59:17.045268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.055227 (+ 9959us) service_pool.cc:225] Handling call
1215 08:59:18.552157 (+1496930us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:18.652495 (+100338us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7540096}
W20251215 08:59:18.600859 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 484211us
I20251215 08:59:18.344627 23122 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:18.640146 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:18.641243 23191 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.460449 22853 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.659287 21408 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:18.661264 22853 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:18.375164 23237 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:18.423695 23193 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:18.437237 23276 raft_consensus.cc:3055] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:18.542697 23284 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.553737 23248 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:18.469985 23027 raft_consensus.cc:697] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:18.603840 23220 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.444221 22961 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:18.678409 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:18.692345 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:18.823304 23220 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.694912 23191 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:18.825245 23220 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.686721 23290 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.825946 23290 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.826114 23027 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:18.697772 23298 log.cc:927] Time spent T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.284s user 0.001s sys 0.000s
I20251215 08:59:18.700812 22799 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:18.648571 23244 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.717892 23310 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:18.722003 23048 consensus_meta.cc:287] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.967s user 0.009s sys 0.000s
I20251215 08:59:18.722486 23157 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.664113 23294 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:18.724167 21263 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1333) took 1281 ms (client timeout 1528 ms). Trace:
W20251215 08:59:18.882476 21263 rpcz_store.cc:269] 1215 08:59:17.442407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.442531 (+ 124us) service_pool.cc:225] Handling call
1215 08:59:18.723977 (+1281446us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b540033ac54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:18.724101 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":244823424}
I20251215 08:59:18.725073 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1349) took 1037 ms. Trace:
I20251215 08:59:18.709249 23195 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:18.719065 23297 log.cc:927] Time spent T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.582s user 0.000s sys 0.001s
I20251215 08:59:18.712882 23207 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:18.713174 23291 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.742314 23292 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:18.750658 23287 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:18.753422 23283 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:18.754492 21266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1345) took 1217 ms. Trace:
I20251215 08:59:18.898072 21266 rpcz_store.cc:276] 1215 08:59:17.537190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.545303 (+ 8113us) service_pool.cc:225] Handling call
1215 08:59:18.754282 (+1208979us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b540028a354. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:18.754419 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":232250240}
I20251215 08:59:18.751853 23212 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.682920 23122 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.765079 21378 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:18.766317 23199 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.772732 23267 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.785035 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:18.911412 21980 raft_consensus.cc:2410] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251215 08:59:18.785602 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.912076 21407 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:18.913309 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.914710 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:18.814221 21264 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.939s user 0.003s sys 0.014s
I20251215 08:59:18.808701 23322 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:18.876442 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.820713 23276 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:18.821323 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.822208 22895 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.883765 21262 rpcz_store.cc:276] 1215 08:59:17.707244 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.707409 (+ 165us) service_pool.cc:225] Handling call
1215 08:59:18.724883 (+1017474us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b54000f7654. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:18.724997 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198880640}
W20251215 08:59:18.886567 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 230950us
I20251215 08:59:18.893000 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:18.824848 23191 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:18.903563 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 138618us
I20251215 08:59:18.928867 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.830147 23237 consensus_queue.cc:1048] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20251215 08:59:18.904881 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:18.936538 21266 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 6 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20251215 08:59:18.936757 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.822016 23248 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:18.885628 23048 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:18.915376 21264 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.040s user 0.004s sys 0.015s
I20251215 08:59:18.834407 22961 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:18.916877 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 311860us
I20251215 08:59:18.886294 23157 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:18.873747 22799 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:18.927517 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:18.903920 23207 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.916450 22938 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:18.885464 23310 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:18.910557 23212 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:18.911652 23291 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.939589 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:18.938359 23256 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:18.941612 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.947474 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 494) took 1109 ms. Trace:
I20251215 08:59:18.962924 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1561) took 1135 ms. Trace:
I20251215 08:59:18.919785 23267 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.948401 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:18.927093 23321 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:18.956022 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:18.963253 21259 rpcz_store.cc:276] 1215 08:59:17.829987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.642199 (+812212us) service_pool.cc:225] Handling call
1215 08:59:18.642382 (+ 183us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:18.642501 (+ 119us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:18.905724 (+263223us) log.cc:844] Serialized 14 byte log entry
1215 08:59:18.962641 (+ 56917us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:18.962690 (+ 49us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:18.962726 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:18.962782 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:18.962879 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":295838,"thread_start_us":319554,"threads_started":2}
I20251215 08:59:18.963224 21978 rpcz_store.cc:276] 1215 08:59:17.838283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.838427 (+ 144us) service_pool.cc:225] Handling call
1215 08:59:17.853891 (+ 15464us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:17.854126 (+ 235us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:17.854153 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:17.854614 (+ 461us) log.cc:844] Serialized 26 byte log entry
1215 08:59:18.041694 (+187080us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:18.041741 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:18.041823 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:18.947210 (+905387us) raft_consensus.cc:1685] finished
1215 08:59:18.947239 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:18.947400 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":186732,"threads_started":1,"wal-append.queue_time_us":301578}
I20251215 08:59:18.969075 21409 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20251215 08:59:18.968830 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1551) took 1222 ms. Trace:
I20251215 08:59:18.970646 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:18.971805 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:18.972369 21978 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:18.973414 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.974152 21407 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:59:18.975078 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.973292 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:18.976949 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.977695 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:18.978325 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.970021 21264 rpcz_store.cc:276] 1215 08:59:17.746439 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.836658 (+ 90219us) service_pool.cc:225] Handling call
1215 08:59:17.836958 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:17.837166 (+ 208us) raft_consensus.cc:1534] Early marking committed up to index 7
1215 08:59:18.945288 (+1108122us) log.cc:844] Serialized 14 byte log entry
1215 08:59:18.968418 (+ 23130us) log.cc:844] Serialized 14 byte log entry
1215 08:59:18.968562 (+ 144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:18.968608 (+ 46us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:18.968641 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:18.968698 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:18.968786 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3365,"prepare.queue_time_us":1065734,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":1089514,"threads_started":2}
I20251215 08:59:18.933894 22895 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.961061 23290 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:18.925588 23322 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.989154 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.991791 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.971874 23315 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:18.951922 23319 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.057s
I20251215 08:59:18.957530 23207 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:19.055339 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 77613.5us
I20251215 08:59:18.978420 23195 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:19.017844 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:19.057520 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.077987 23325 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.088421 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:19.056341 21259 raft_consensus.cc:1240] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:19.080118 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:18.965981 23291 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.090869 21407 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:19.092473 21407 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2c10c3de775846868774c98cc0a75094"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:19.093492 21407 raft_consensus.cc:606] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:19.080329 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:19.053275 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 71629us
I20251215 08:59:18.957571 23157 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:18.968937 23212 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:18.990350 23326 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:18.968514 23256 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.063500 23237 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:19.083698 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:19.083945 23214 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.054966 23220 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:18.976418 23267 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.096177 21304 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:19.096413 21980 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:19.055994 23228 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:19.098170 21307 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:19.104619 21259 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:19.095153 23315 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:19.089964 22938 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:19.115226 23267 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.115751 23267 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.084421 23318 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:19.112087 21259 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:19.110544 23237 raft_consensus.cc:697] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:18.972206 23276 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.113632 23322 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:19.113680 23323 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
I20251215 08:59:18.971873 23321 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.114606 23200 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:19.102536 23191 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.116199 23267 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.102569 22799 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.103122 23328 consensus_queue.cc:1059] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20251215 08:59:19.110255 23220 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.134183 23328 raft_consensus.cc:3055] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:19.134564 23328 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:19.135499 23212 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.107527 23326 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.111971 23122 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.117873 23315 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:19.124377 23268 consensus_meta.cc:287] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.968s user 0.011s sys 0.000s
I20251215 08:59:19.109128 23214 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.116219 23291 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.120992 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.104555 23195 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:19.125399 23244 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.135510 23220 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.115226 22938 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.132393 23319 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:19.163445 22961 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.136933 23276 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:19.116289 23228 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.136173 23200 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.168020 23268 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.145301 23326 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:19.159648 23334 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:19.160713 23315 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:19.181099 23214 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.135398 23328 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.099962 23325 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.173822 23195 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.131549 23318 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.177188 21411 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251215 08:59:19.188666 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:19.189759 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.183629 22895 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.179797 23248 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.181933 23319 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.204397 23157 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:19.207249 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.178933 23180 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.195194 23195 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:19.194144 23207 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.194795 23214 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.206032 23319 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:19.205770 23314 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:19.205034 21972 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 369) took 1194 ms. Trace:
I20251215 08:59:19.220038 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.209940 23328 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:19.226711 23341 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.227495 23341 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.230614 23341 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.183804 23237 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.206746 23334 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:19.232399 23334 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.226109 23338 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.234272 23338 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.236958 23338 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.241477 22895 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.242143 22895 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.257728 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:19.259723 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:19.278623 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 180565us
I20251215 08:59:19.279577 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:19.245353 22895 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.283633 21259 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251215 08:59:19.283630 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:19.263769 23228 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.277415 23014 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.224956 23310 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.289824 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.290426 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.290949 21334 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251215 08:59:19.282761 23256 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.283138 23248 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:19.293068 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.315747 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.316504 21334 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:19.317476 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:19.297544 21340 raft_consensus.cc:1240] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:19.223719 23214 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.280125 23326 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.286540 23267 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.318077 21334 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:19.293318 23332 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20251215 08:59:19.325331 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:19.293058 23340 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.279891 23314 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.292904 23243 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.310722 22938 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.325017 23193 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.333472 23337 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.333830 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.334100 23268 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.334889 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:19.341133 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:19.389322 21410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59102 (request call id 151) took 2163 ms (client timeout 2063 ms). Trace:
I20251215 08:59:19.389515 21259 raft_consensus.cc:2410] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20251215 08:59:19.389750 21410 rpcz_store.cc:269] 1215 08:59:17.226358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.228592 (+ 2234us) service_pool.cc:225] Handling call
1215 08:59:19.334699 (+2106107us) spinlock_profiling.cc:243] Waited 5.98 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:19.389230 (+ 54531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13750144}
I20251215 08:59:19.390169 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.390659 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.391367 21259 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:19.312891 23014 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.319248 23248 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:19.348891 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.391682 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.392907 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.394626 21337 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:19.318961 23328 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.397289 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:19.387253 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.400458 21335 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:19.402457 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.396651 23248 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:19.403445 23328 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.421247 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.423094 21266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1564) took 1569 ms. Trace:
I20251215 08:59:19.424713 21337 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:19.424569 21266 rpcz_store.cc:276] 1215 08:59:17.853387 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.937215 (+1083828us) service_pool.cc:225] Handling call
1215 08:59:18.937561 (+ 346us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:18.937908 (+ 347us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:18.937958 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:18.944702 (+ 6744us) log.cc:844] Serialized 26 byte log entry
1215 08:59:19.055406 (+110704us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:19.055440 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:19.055497 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:19.422893 (+367396us) raft_consensus.cc:1685] finished
1215 08:59:19.422912 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:19.423038 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":558,"raft.queue_time_us":98,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":8944,"thread_start_us":88749,"threads_started":1,"wal-append.queue_time_us":158241}
W20251215 08:59:19.427222 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 94610.5us
I20251215 08:59:19.428220 23325 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.399906 23268 consensus_queue.cc:1059] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20251215 08:59:19.435624 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.437067 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.413555 23321 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.441197 23326 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.400910 23340 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.440369 23314 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.444530 21334 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:19.396800 23243 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.410130 23343 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:19.446372 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:19.438457 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.225397 21972 rpcz_store.cc:276] 1215 08:59:18.010191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.270723 (+260532us) service_pool.cc:225] Handling call
1215 08:59:18.630180 (+359457us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:18.903084 (+272904us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:18.910238 (+ 7154us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:18.984350 (+ 74112us) log.cc:844] Serialized 26 byte log entry
1215 08:59:19.113395 (+129045us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:19.113434 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:19.113512 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:19.189605 (+ 76093us) raft_consensus.cc:1685] finished
1215 08:59:19.191574 (+ 1969us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:19.195505 (+ 3931us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22159,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":2338,"thread_start_us":263220,"threads_started":2,"wal-append.queue_time_us":158903}
I20251215 08:59:19.454126 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.452546 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.432963 23318 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.455003 21334 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:19.445312 21411 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251215 08:59:19.455353 23343 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:19.451073 23344 log.cc:927] Time spent T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20251215 08:59:19.461539 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.400070 23337 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:19.469276 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.466651 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.472188 23328 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:19.530782 21412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1527) took 2238 ms (client timeout 1915 ms). Trace:
I20251215 08:59:19.537886 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.558144 21341 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:19.535506 21412 rpcz_store.cc:269] 1215 08:59:17.292304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.357220 (+ 64916us) service_pool.cc:225] Handling call
1215 08:59:19.530605 (+2173385us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5400341054. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:19.530712 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25291136}
I20251215 08:59:19.478910 23193 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.473460 23337 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.484078 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:19.575742 21381 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:19.574177 21412 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:19.576310 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.579349 21259 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:19.487212 23325 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:19.573658 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 117217us
I20251215 08:59:19.531949 23262 raft_consensus.cc:3055] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:19.488196 23214 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.588395 23325 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.600097 21946 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:19.603117 23246 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:19.603530 23246 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:19.603853 23246 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:19.605567 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.606204 21259 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:19.606978 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.607952 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.610221 23254 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:19.610636 23254 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:19.610991 23254 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:19.633086 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:19.684572 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.753607 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.754881 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.584725 21411 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:19.584838 23262 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:19.800809 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:19.812176 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:19.815433 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.815645 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:19.816484 21259 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:19.590803 23328 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:19.766367 23268 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.582305 23315 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:19.850703 21232 leader_election.cc:336] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.752s (SENT)
I20251215 08:59:19.891640 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:19.869267 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.618427 23340 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.756709 23343 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:19.888477 21230 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.219s (SENT) [suppressed 31 similar messages]
I20251215 08:59:19.583451 23319 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.831795 23318 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:19.611189 23246 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:19.894649 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.907238 21334 raft_consensus.cc:2393] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:19.868233 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:19.841563 23325 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:19.911027 21336 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:19.908063 23318 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.915030 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46170 (request call id 372) took 1089 ms. Trace:
I20251215 08:59:19.919099 21980 rpcz_store.cc:276] 1215 08:59:18.825377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.926718 (+101341us) service_pool.cc:225] Handling call
1215 08:59:19.914817 (+988099us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54003e08d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:19.914959 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":317781888}
I20251215 08:59:19.920498 23318 leader_election.cc:290] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.920604 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:19.913532 23246 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.924093 23319 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.957063 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1146) took 2738 ms. Trace:
I20251215 08:59:19.957545 21332 rpcz_store.cc:276] 1215 08:59:17.219015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.219326 (+ 311us) service_pool.cc:225] Handling call
1215 08:59:19.956463 (+2737137us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398cb6d58e 00007f398610c437 00007f398610c3c2 00007f398610c2f1 00007f398610c26c 00007f398610b522 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:19.956954 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":771010432}
I20251215 08:59:19.926311 23149 raft_consensus.cc:3055] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:19.927902 21257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1558) took 2111 ms. Trace:
W20251215 08:59:19.929970 21979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46170 (request call id 361) took 2620 ms (client timeout 1577 ms). Trace:
W20251215 08:59:19.971544 21979 rpcz_store.cc:269] 1215 08:59:17.309237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.309370 (+ 133us) service_pool.cc:225] Handling call
1215 08:59:19.929715 (+2620345us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54003f02d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:19.929887 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1786436480}
W20251215 08:59:19.930791 21975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 438) took 2697 ms (client timeout 2224 ms). Trace:
I20251215 08:59:19.932996 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1533) took 1149 ms. Trace:
I20251215 08:59:19.836694 23243 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:19.938076 23086 raft_consensus.cc:3055] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:19.942147 21980 raft_consensus.cc:2410] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251215 08:59:19.952529 21976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 435) took 2938 ms. Trace:
I20251215 08:59:19.959448 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:19.904181 23343 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:19.976696 21409 rpcz_store.cc:276] 1215 08:59:18.783732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.783955 (+ 223us) service_pool.cc:225] Handling call
1215 08:59:18.784341 (+ 386us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:18.969471 (+185130us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:19.932680 (+963209us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54000d16d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:19.932922 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424256384}
I20251215 08:59:19.934983 21974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 491) took 2588 ms. Trace:
I20251215 08:59:19.943015 21972 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:19.836966 23328 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.981967 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:19.981921 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:19.943209 23325 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:19.895393 23268 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.968000 23149 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:19.983574 21976 rpcz_store.cc:276] 1215 08:59:17.014509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.016213 (+ 1704us) service_pool.cc:225] Handling call
1215 08:59:17.037992 (+ 21779us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:17.072952 (+ 34960us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:17.074231 (+ 1279us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:17.182598 (+108367us) log.cc:844] Serialized 52 byte log entry
1215 08:59:17.311565 (+128967us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:17.378724 (+ 67159us) log.cc:844] Serialized 14 byte log entry
1215 08:59:17.411511 (+ 32787us) log.cc:844] Serialized 14 byte log entry
1215 08:59:17.435134 (+ 23623us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:19.938622 (+2503488us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b18003fd420. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ad3fe6 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49
1215 08:59:19.939349 (+ 727us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b54004b19d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:19.939423 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:19.942419 (+ 2996us) raft_consensus.cc:1685] finished
1215 08:59:19.952294 (+ 9875us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:19.952466 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13930,"prepare.queue_time_us":72233,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"raft.queue_time_us":10644,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":33106,"spinlock_wait_cycles":62529280,"thread_start_us":182424,"threads_started":2,"wal-append.queue_time_us":196753,"wal-append.run_cpu_time_us":3050,"wal-append.run_wall_time_us":1165154}
I20251215 08:59:19.998445 23149 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:19.999446 23149 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:19.999912 23236 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:19.976513 21975 rpcz_store.cc:269] 1215 08:59:17.233777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.241306 (+ 7529us) service_pool.cc:225] Handling call
1215 08:59:19.930546 (+2689240us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b540039c1d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:19.930711 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3969920}
W20251215 08:59:19.986289 23347 log.cc:927] Time spent T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.363s user 0.000s sys 0.001s
W20251215 08:59:19.992499 21232 leader_election.cc:336] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.894s (SENT)
I20251215 08:59:19.995062 21980 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:20.012118 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 502) took 1020 ms. Trace:
I20251215 08:59:20.013192 21980 rpcz_store.cc:276] 1215 08:59:18.992194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.981530 (+989336us) service_pool.cc:225] Handling call
1215 08:59:20.012058 (+ 30528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.000507 23236 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:19.980430 23086 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:20.014669 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.014832 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.078747 21975 raft_consensus.cc:2410] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20251215 08:59:20.079135 23328 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:20.079289 23268 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:19.996663 23325 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:19.971307 21257 rpcz_store.cc:276] 1215 08:59:17.819648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.155677 (+336029us) service_pool.cc:225] Handling call
1215 08:59:18.736401 (+580724us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:18.741126 (+ 4725us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:18.745949 (+ 4823us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:18.746014 (+ 65us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:18.746055 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:18.746129 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:19.927640 (+1181511us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54000f7650. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:19.927809 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1045441792}
W20251215 08:59:20.012830 21232 leader_election.cc:336] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.915s (SENT)
I20251215 08:59:20.016711 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:19.992133 23248 consensus_meta.cc:287] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.563s user 0.008s sys 0.002s
W20251215 08:59:19.990470 23195 consensus_meta.cc:287] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.769s user 0.005s sys 0.005s
I20251215 08:59:19.999136 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:19.837195 23353 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.624s
I20251215 08:59:19.989370 23343 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:20.079946 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.079710 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:20.151559 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.152098 21972 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:20.152557 23086 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:20.153577 21232 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:20.167279 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 154474us
W20251215 08:59:20.136466 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.942145 23262 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:20.077309 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.166615 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 371) took 1697 ms. Trace:
I20251215 08:59:20.176342 21978 rpcz_store.cc:276] 1215 08:59:18.469569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.983156 (+513587us) service_pool.cc:225] Handling call
1215 08:59:18.983481 (+ 325us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:19.205455 (+221974us) raft_consensus.cc:1534] Early marking committed up to index 5
1215 08:59:19.206160 (+ 705us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:19.206664 (+ 504us) log.cc:844] Serialized 26 byte log entry
1215 08:59:19.422082 (+215418us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:19.422123 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.166209 (+744086us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540066bf54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.166256 (+ 47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:20.166306 (+ 50us) raft_consensus.cc:1685] finished
1215 08:59:20.166321 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.166552 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21522,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"spinlock_wait_cycles":295752064,"thread_start_us":279553,"threads_started":2,"wal-append.queue_time_us":234586}
I20251215 08:59:20.178814 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.179566 21978 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:20.180670 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.181335 21978 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:20.172731 23236 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:20.184960 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.185434 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.186092 21980 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:20.186555 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 501) took 1194 ms. Trace:
I20251215 08:59:20.186851 21980 rpcz_store.cc:276] 1215 08:59:18.991801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.172975 (+1181174us) service_pool.cc:225] Handling call
1215 08:59:20.186501 (+ 13526us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.187269 21979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 445) took 1251 ms. Trace:
I20251215 08:59:20.188282 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 498) took 1226 ms. Trace:
I20251215 08:59:20.187742 21979 rpcz_store.cc:276] 1215 08:59:18.935786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.186267 (+1250481us) service_pool.cc:225] Handling call
1215 08:59:20.186635 (+ 368us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.186852 (+ 217us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.186889 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.186947 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:20.186991 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.187061 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.187203 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.190239 23343 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:20.175493 23254 consensus_meta.cc:287] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.564s user 0.000s sys 0.011s
I20251215 08:59:20.190510 21979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 499) took 1221 ms. Trace:
I20251215 08:59:20.172727 21975 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:20.198346 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1474) took 2235 ms. Trace:
I20251215 08:59:20.198894 21338 rpcz_store.cc:276] 1215 08:59:17.962926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.963042 (+ 116us) service_pool.cc:225] Handling call
1215 08:59:17.989907 (+ 26865us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:18.059671 (+ 69764us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:18.059702 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:18.059740 (+ 38us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:18.059777 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.198042 (+2138265us) spinlock_profiling.cc:243] Waited 69.6 ms on lock 0x7b5400153e54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.198105 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.198274 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":160059648}
I20251215 08:59:20.193485 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 444) took 1261 ms. Trace:
I20251215 08:59:20.192473 21980 rpcz_store.cc:276] 1215 08:59:18.961606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.187478 (+1225872us) service_pool.cc:225] Handling call
1215 08:59:20.187761 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.187932 (+ 171us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.187978 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.188021 (+ 43us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:20.188054 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.188122 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.188231 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.200731 21978 rpcz_store.cc:276] 1215 08:59:18.931961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.192714 (+1260753us) service_pool.cc:225] Handling call
1215 08:59:20.192978 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.193163 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.193189 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.193231 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:20.193264 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.193332 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.193436 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.191296 21979 rpcz_store.cc:276] 1215 08:59:18.968780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.189359 (+1220579us) service_pool.cc:225] Handling call
1215 08:59:20.189708 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.190097 (+ 389us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.190144 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.190198 (+ 54us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:20.190238 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.190306 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.190445 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.191186 23254 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:20.194377 23231 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:20.202520 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 503) took 1210 ms. Trace:
I20251215 08:59:20.211534 21978 rpcz_store.cc:276] 1215 08:59:18.992539 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.201691 (+1209152us) service_pool.cc:225] Handling call
1215 08:59:20.202005 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.202182 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.202216 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.202267 (+ 51us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:20.202306 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.202375 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.202468 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:20.196914 23276 consensus_meta.cc:287] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.012s user 0.011s sys 0.000s
I20251215 08:59:20.204380 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.235369 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.224107 21332 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:20.237290 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 500) took 1246 ms. Trace:
I20251215 08:59:20.237820 21980 rpcz_store.cc:276] 1215 08:59:18.991381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.201585 (+1210204us) service_pool.cc:225] Handling call
1215 08:59:20.219227 (+ 17642us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.236759 (+ 17532us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.236803 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.236912 (+ 109us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:20.236956 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.237057 (+ 101us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.237223 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.217324 23254 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:20.239588 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.240103 21332 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:20.240270 21979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 374) took 1246 ms. Trace:
I20251215 08:59:20.167753 23248 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:20.155045 23195 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:20.240562 21979 rpcz_store.cc:276] 1215 08:59:18.993479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.204365 (+1210886us) service_pool.cc:225] Handling call
1215 08:59:20.239815 (+ 35450us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.239975 (+ 160us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.239997 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.240031 (+ 34us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:20.240055 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.240102 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.240224 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:20.217741 21410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s user 0.026s sys 0.066s
W20251215 08:59:20.248906 21265 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1342) took 2772 ms (client timeout 1666 ms). Trace:
I20251215 08:59:20.203538 23315 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:20.200269 21972 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 505) took 1049 ms. Trace:
I20251215 08:59:20.245662 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.245869 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.252418 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.252547 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.253000 21332 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:20.253293 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.253595 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.253903 21335 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:20.243359 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 447) took 1124 ms. Trace:
W20251215 08:59:20.249367 21265 rpcz_store.cc:269] 1215 08:59:17.469705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.501210 (+ 31505us) service_pool.cc:225] Handling call
1215 08:59:20.242290 (+2741080us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540033a254. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:20.242393 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627701120}
I20251215 08:59:20.254385 21338 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
W20251215 08:59:20.257372 21337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.014s sys 0.016s
W20251215 08:59:20.257752 21337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s user 0.014s sys 0.016s
I20251215 08:59:20.258764 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.259518 21334 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:20.260490 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.262457 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 148) took 2427 ms. Trace:
I20251215 08:59:20.262944 21262 rpcz_store.cc:276] 1215 08:59:17.834909 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.932803 (+1097894us) service_pool.cc:225] Handling call
1215 08:59:18.946136 (+ 13333us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:18.967562 (+ 21426us) raft_consensus.cc:1534] Early marking committed up to index 7
1215 08:59:18.971906 (+ 4344us) log.cc:844] Serialized 14 byte log entry
1215 08:59:19.055187 (+ 83281us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:19.055238 (+ 51us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:19.055277 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.207318 (+1152041us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b18003bc7a0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:20.262180 (+ 54862us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5400134cd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.262246 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.262391 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41756,"raft.queue_time_us":37848,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":22872,"spinlock_wait_cycles":58046976,"thread_start_us":45219,"threads_started":1,"wal-append.queue_time_us":112591}
I20251215 08:59:20.268242 23048 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:20.271958 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34992 (request call id 20) took 1350 ms. Trace:
I20251215 08:59:20.275183 23189 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:20.280356 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1556) took 2523 ms. Trace:
I20251215 08:59:20.280841 21260 rpcz_store.cc:276] 1215 08:59:17.756780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.020928 (+264148us) service_pool.cc:225] Handling call
1215 08:59:18.021250 (+ 322us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:18.271647 (+250397us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:18.271675 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:18.271713 (+ 38us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:18.271742 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.279979 (+2008237us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b5400170a54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.280040 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.280239 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":575414016}
W20251215 08:59:20.283267 23027 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.186s user 0.010s sys 0.015s
W20251215 08:59:20.284197 23027 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.193s user 0.010s sys 0.016s
I20251215 08:59:20.284015 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1531) took 2900 ms. Trace:
I20251215 08:59:20.285946 21408 rpcz_store.cc:276] 1215 08:59:17.383853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.821193 (+437340us) service_pool.cc:225] Handling call
1215 08:59:17.855557 (+ 34364us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:18.659713 (+804156us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.283717 (+1624004us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b54001dfbd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.283914 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1847800192}
W20251215 08:59:20.250790 21410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.818s user 0.026s sys 0.066s
I20251215 08:59:20.272462 21407 rpcz_store.cc:276] 1215 08:59:18.921996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.092232 (+170236us) service_pool.cc:225] Handling call
1215 08:59:20.271575 (+1179343us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b540023a8a0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987b31550 00007f3987bae2d2 00007f398cb6c13a 00007f3986107934 00007f39861078c2 00007f39861077f1 00007f398610776c 00007f3986106a22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.271884 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23627648}
I20251215 08:59:20.288987 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:20.289577 21332 raft_consensus.cc:2376] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4. Re-sending same reply.
I20251215 08:59:20.289594 21333 raft_consensus.cc:2393] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:20.290349 21979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 448) took 1040 ms. Trace:
I20251215 08:59:20.293370 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.293788 21979 rpcz_store.cc:276] 1215 08:59:19.250236 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.248786 (+998550us) service_pool.cc:225] Handling call
1215 08:59:20.289902 (+ 41116us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.290052 (+ 150us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.290073 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.290111 (+ 38us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:20.290156 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.290213 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.290301 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:20.294345 21266 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.014s sys 0.037s
W20251215 08:59:20.294728 21266 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s user 0.014s sys 0.037s
I20251215 08:59:20.292865 21334 raft_consensus.cc:2393] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:20.302881 21381 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s user 0.024s sys 0.052s
I20251215 08:59:20.255271 21978 rpcz_store.cc:276] 1215 08:59:19.119112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.212712 (+1093600us) service_pool.cc:225] Handling call
1215 08:59:20.234619 (+ 21907us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.242999 (+ 8380us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.243031 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.243099 (+ 68us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:20.243127 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.243184 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.243298 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:20.305719 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.256780 23262 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:20.308589 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:20.306504 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:19.986686 21974 rpcz_store.cc:276] 1215 08:59:17.346793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.348072 (+ 1279us) service_pool.cc:225] Handling call
1215 08:59:19.933495 (+2585423us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:19.933767 (+ 272us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:19.933809 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:19.933870 (+ 61us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:19.933955 (+ 85us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:19.934048 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:19.934736 (+ 688us) spinlock_profiling.cc:243] Waited 2.55 s on lock 0x7b540039c1d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:19.934878 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5872261888}
I20251215 08:59:20.310811 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:20.311190 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.311563 21265 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:20.311996 21338 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
W20251215 08:59:20.305893 21341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.014s sys 0.004s
W20251215 08:59:20.314986 21341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s user 0.015s sys 0.005s
I20251215 08:59:20.317975 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.318426 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.319157 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.319885 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.320165 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.321357 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.321408 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.321276 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:20.322115 21407 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:20.322571 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.322862 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.323170 21412 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:59:20.323828 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.324102 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.324618 21412 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:20.324848 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.325783 21408 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 6 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20251215 08:59:20.287482 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:20.329420 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.309465 21335 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:20.330225 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.251245 21972 rpcz_store.cc:276] 1215 08:59:19.143257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.001251 (+857994us) service_pool.cc:225] Handling call
1215 08:59:20.192203 (+190952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.289170 21980 raft_consensus.cc:2955] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:20.300971 21973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 443) took 2858 ms. Trace:
I20251215 08:59:20.334132 21979 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20251215 08:59:20.338222 21979 pending_rounds.cc:85] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Aborting all ops after (but not including) 4
I20251215 08:59:20.338122 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.338708 21979 pending_rounds.cc:107] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251215 08:59:20.339020 21979 raft_consensus.cc:2889] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:20.340737 21409 raft_consensus.cc:3055] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:20.340833 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.341383 21409 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:20.341711 21408 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:20.339131 21338 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
W20251215 08:59:20.307420 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.344714 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.316730 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:20.346647 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.353358 21379 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:20.349196 21409 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:20.358457 21409 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:20.350049 23333 log.cc:927] Time spent T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20251215 08:59:20.361285 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.362711 21332 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:20.362949 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.363543 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.360246 21265 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:20.363525 21338 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
W20251215 08:59:20.367898 21380 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.002s sys 0.017s
I20251215 08:59:20.350943 23212 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:20.368971 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.368991 21380 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.531s user 0.002s sys 0.017s
I20251215 08:59:20.370679 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1555) took 1044 ms. Trace:
I20251215 08:59:20.372131 21338 rpcz_store.cc:276] 1215 08:59:19.325935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.369885 (+1043950us) service_pool.cc:225] Handling call
1215 08:59:20.370212 (+ 327us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.370395 (+ 183us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.370419 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.370458 (+ 39us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:20.370482 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.370535 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.370634 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:20.375171 21229 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.000s sys 0.021s
W20251215 08:59:20.375496 21415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1517) took 3148 ms (client timeout 2122 ms). Trace:
W20251215 08:59:20.375561 21229 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.000s sys 0.021s
I20251215 08:59:20.375715 21414 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:59:20.375936 21381 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:20.361716 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:20.375976 21415 rpcz_store.cc:269] 1215 08:59:17.227307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.227431 (+ 124us) service_pool.cc:225] Handling call
1215 08:59:20.375281 (+3147850us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b540034f654. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:20.375432 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":381234176}
W20251215 08:59:20.379194 21944 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.003s sys 0.015s
W20251215 08:59:20.379611 21944 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.003s sys 0.015s
W20251215 08:59:20.380273 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.386317 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.360836 23150 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:20.393110 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.362190 23325 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:20.395298 21232 leader_election.cc:336] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 2.260s (SENT)
I20251215 08:59:20.394742 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:20.396685 21265 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:20.398295 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.398854 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.370944 23351 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.042s sys 0.042s
I20251215 08:59:20.399645 21229 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:20.373416 23212 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:20.402165 21229 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:20.388883 23321 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.792s user 0.017s sys 0.025s
I20251215 08:59:20.403280 21338 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251215 08:59:20.413486 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:20.401064 23149 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:20.405263 21304 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.027s sys 0.032s
I20251215 08:59:20.416460 21229 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:20.401693 23351 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.799s user 0.043s sys 0.042s
I20251215 08:59:20.409221 21232 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:20.412258 21265 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:20.398132 23240 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:20.397996 22927 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:20.420620 21946 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:20.444523 21332 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20251215 08:59:20.416611 21304 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.027s sys 0.032s
I20251215 08:59:20.420920 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.446250 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:20.447747 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1559) took 1025 ms. Trace:
I20251215 08:59:20.449345 21332 rpcz_store.cc:276] 1215 08:59:19.421868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.401560 (+979692us) service_pool.cc:225] Handling call
1215 08:59:20.421324 (+ 19764us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:20.447423 (+ 26099us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.447677 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.450042 21979 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:20.420738 23149 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:20.483235 23149 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:20.484016 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1550) took 1578 ms. Trace:
I20251215 08:59:20.484463 21410 rpcz_store.cc:276] 1215 08:59:18.905324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.390598 (+485274us) service_pool.cc:225] Handling call
1215 08:59:19.390945 (+ 347us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:19.391176 (+ 231us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:19.391203 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:19.394109 (+ 2906us) log.cc:844] Serialized 26 byte log entry
1215 08:59:20.289041 (+894932us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:20.292351 (+ 3310us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.306253 (+ 13902us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:20.483782 (+177529us) raft_consensus.cc:1685] finished
1215 08:59:20.483805 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.483955 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":858408,"threads_started":1,"wal-append.queue_time_us":983993}
I20251215 08:59:20.480844 21975 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:20.446460 23221 raft_consensus.cc:2764] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251215 08:59:20.342239 21973 rpcz_store.cc:276] 1215 08:59:17.443019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.447809 (+ 4790us) service_pool.cc:225] Handling call
1215 08:59:17.453761 (+ 5952us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:17.715778 (+262017us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.284754 (+2568976us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b54005febd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.292372 (+ 7618us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102111488}
I20251215 08:59:20.463984 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:20.413292 23212 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:20.465003 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.559732 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.415046 23248 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:20.466507 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 45968.9us
I20251215 08:59:20.418292 23195 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:20.412947 23321 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.823s user 0.017s sys 0.025s
I20251215 08:59:20.420549 23060 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:20.576077 23060 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:20.576503 23060 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:20.414624 22895 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:20.479728 21378 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.000s sys 0.017s
W20251215 08:59:20.577378 21378 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.000s sys 0.017s
I20251215 08:59:20.445943 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:20.580411 23129 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:20.531550 21972 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:20.532824 23178 raft_consensus.cc:3055] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:20.451658 23251 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251215 08:59:20.533951 23363 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:20.539666 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1553) took 1220 ms. Trace:
W20251215 08:59:20.612017 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.450448 23209 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.023s sys 0.011s
I20251215 08:59:20.445942 23048 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:20.566131 21946 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:20.567680 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:20.562913 23357 log.cc:927] Time spent T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20251215 08:59:20.572515 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1554) took 1248 ms. Trace:
I20251215 08:59:20.577308 22895 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:20.522179 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1560) took 1084 ms. Trace:
I20251215 08:59:20.419097 23325 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.133s
I20251215 08:59:20.588946 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.590539 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.596737 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.597754 21979 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:20.598390 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:20.592907 23178 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:20.602025 21415 raft_consensus.cc:606] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:20.594027 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 50556.4us
I20251215 08:59:20.606101 21337 rpcz_store.cc:276] 1215 08:59:19.319717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.470393 (+150676us) service_pool.cc:225] Handling call
1215 08:59:19.557975 (+ 87582us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:19.559964 (+ 1989us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:19.560006 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:19.560592 (+ 586us) log.cc:844] Serialized 26 byte log entry
1215 08:59:20.258206 (+697614us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:20.258263 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.258341 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:20.539147 (+280806us) raft_consensus.cc:1685] finished
1215 08:59:20.539433 (+ 286us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.539596 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1447,"raft.queue_time_us":3390,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":3155,"thread_start_us":676078,"threads_started":1,"wal-append.queue_time_us":828641}
W20251215 08:59:20.650348 21944 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.650s (SENT)
W20251215 08:59:20.657660 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 91610.1us
I20251215 08:59:20.656936 21338 rpcz_store.cc:276] 1215 08:59:19.438204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.415778 (+977574us) service_pool.cc:225] Handling call
1215 08:59:20.466034 (+ 50256us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.473175 (+ 7141us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.478383 (+ 5208us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.478455 (+ 72us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:20.478507 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.478594 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.522090 (+ 43496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:20.673527 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:20.673986 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 57281.3us
I20251215 08:59:20.674968 21415 raft_consensus.cc:614] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:20.676240 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:20.660286 21333 raft_consensus.cc:2393] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:20.646152 21341 rpcz_store.cc:276] 1215 08:59:19.324184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.573602 (+249418us) service_pool.cc:225] Handling call
1215 08:59:19.574053 (+ 451us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:19.574258 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:19.574310 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:19.574798 (+ 488us) log.cc:844] Serialized 26 byte log entry
1215 08:59:20.315483 (+740685us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:20.315542 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.315631 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:20.571219 (+255588us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b240001f388. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f398495a9a4 00007f3984958aab 00007f39849e79a9 00007f39849e7635 00007f39849e6c7d 00007f39877374ae 00007f398774ec2c 00007f398774df56 00007f398774db78 00007f39877575c1
1215 08:59:20.572225 (+ 1006us) raft_consensus.cc:1685] finished
1215 08:59:20.572248 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.572452 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7025920,"thread_start_us":737291,"threads_started":1,"wal-append.queue_time_us":840342}
W20251215 08:59:20.661082 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 176771us
I20251215 08:59:20.674724 21378 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:20.677884 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.679311 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.614941 23240 raft_consensus.cc:697] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:20.681146 21378 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:20.619375 23209 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s user 0.023s sys 0.012s
I20251215 08:59:20.688925 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.682911 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.661319 22895 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:20.682973 23131 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:20.683043 22927 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:20.692652 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.693163 22927 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:20.693822 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:20.694037 23371 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:20.694392 21337 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:20.694569 23371 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:20.695151 22927 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:20.696087 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.696534 23371 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:20.697419 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.698103 21341 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:20.698766 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.700938 21338 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:20.702396 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1539) took 1891 ms. Trace:
I20251215 08:59:20.702739 21413 rpcz_store.cc:276] 1215 08:59:18.810864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.106896 (+296032us) service_pool.cc:225] Handling call
1215 08:59:19.122875 (+ 15979us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:19.188148 (+ 65273us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:19.188208 (+ 60us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:19.188263 (+ 55us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:19.188304 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.702190 (+1513886us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b540025fb54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.702229 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.702339 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":149490048}
W20251215 08:59:20.705238 21406 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1535) took 1920 ms (client timeout 1555 ms). Trace:
I20251215 08:59:20.705435 21977 raft_consensus.cc:2410] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20251215 08:59:20.705669 21406 rpcz_store.cc:269] 1215 08:59:18.791509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.801945 (+ 10436us) service_pool.cc:225] Handling call
1215 08:59:20.590374 (+1788429us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:20.705161 (+114787us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7997696}
I20251215 08:59:20.706334 23267 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:20.706808 23267 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:20.707945 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1452) took 4592 ms. Trace:
I20251215 08:59:20.622210 23048 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:20.688454 23197 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:20.651613 23363 raft_consensus.cc:697] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:20.657712 21976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 507) took 1465 ms. Trace:
I20251215 08:59:20.725199 21266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 161) took 1259 ms. Trace:
W20251215 08:59:20.726603 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 45560.7us
I20251215 08:59:20.716673 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.662348 23027 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:20.727617 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:20.725358 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:20.708318 21339 rpcz_store.cc:276] 1215 08:59:16.115593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:16.115731 (+ 138us) service_pool.cc:225] Handling call
1215 08:59:16.126301 (+ 10570us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:17.445729 (+1319428us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.707634 (+3261905us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400154354. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.707855 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3011600768}
I20251215 08:59:20.725786 21266 rpcz_store.cc:276] 1215 08:59:19.465820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.465965 (+ 145us) service_pool.cc:225] Handling call
1215 08:59:19.466360 (+ 395us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:19.466630 (+ 270us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:19.466669 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:19.467490 (+ 821us) log.cc:844] Serialized 26 byte log entry
1215 08:59:20.295139 (+827649us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:20.295195 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.295287 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:20.724833 (+429546us) raft_consensus.cc:1685] finished
1215 08:59:20.724866 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.725115 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":757630,"threads_started":1,"wal-append.queue_time_us":951680}
I20251215 08:59:20.734262 21409 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:20.734792 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:20.736967 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.679443 23370 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:20.680222 23231 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:20.615870 23190 consensus_queue.cc:1059] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20251215 08:59:20.769716 21408 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:59:20.740617 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.741978 21339 raft_consensus.cc:1240] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:20.790449 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.781778 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:20.773157 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 82838.1us
W20251215 08:59:20.773588 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 46108.2us
I20251215 08:59:20.800841 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:20.801692 21266 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:20.719640 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 363) took 3033 ms. Trace:
I20251215 08:59:20.732955 23048 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:20.771152 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.738786 23262 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:20.814525 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:20.820858 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:20.813895 21333 raft_consensus.cc:1240] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
W20251215 08:59:20.810645 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:20.817000 21261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1335) took 3373 ms (client timeout 1744 ms). Trace:
W20251215 08:59:20.845381 21261 rpcz_store.cc:269] 1215 08:59:17.443996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.444108 (+ 112us) service_pool.cc:225] Handling call
1215 08:59:20.816744 (+3372636us) spinlock_profiling.cc:243] Waited 57.2 ms on lock 0x7b5400358754. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:20.816914 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":131526912}
W20251215 08:59:20.847733 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 171550us
I20251215 08:59:20.719812 23351 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:20.789086 21304 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:20.807929 23197 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:20.786715 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.781484 23231 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:20.821710 23262 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:20.724664 21976 rpcz_store.cc:276] 1215 08:59:19.188103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.243999 (+1055896us) service_pool.cc:225] Handling call
1215 08:59:20.289665 (+ 45666us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.313512 (+ 23847us) raft_consensus.cc:1534] Early marking committed up to index 6
1215 08:59:20.533338 (+219826us) log.cc:844] Serialized 14 byte log entry
1215 08:59:20.573885 (+ 40547us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.592674 (+ 18789us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:20.604398 (+ 11724us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.616793 (+ 12395us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.652109 (+ 35316us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6633,"prepare.queue_time_us":24957,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":3906,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":5795,"thread_start_us":157829,"threads_started":2}
I20251215 08:59:20.806783 23193 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:20.812160 21971 rpcz_store.cc:276] 1215 08:59:17.683904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.701838 (+ 17934us) service_pool.cc:225] Handling call
1215 08:59:17.756213 (+ 54375us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:18.044612 (+288399us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:18.044645 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:18.044688 (+ 43us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:18.044723 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.684285 (+2639562us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54003fe3d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:20.686640 (+ 2355us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:20.717459 (+ 30819us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":621680000}
I20251215 08:59:20.861419 23262 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:20.857440 23369 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:20.842896 21944 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.062s (SENT)
I20251215 08:59:20.843868 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.850988 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:20.851325 21337 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
W20251215 08:59:20.853366 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 64391.4us
I20251215 08:59:20.854143 23197 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:20.833423 21410 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:20.856897 23351 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:20.880271 21946 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:20.740394 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.883253 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.822840 23240 consensus_queue.cc:237] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:20.739847 23178 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:20.838074 21381 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:20.880964 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.833099 23363 consensus_queue.cc:237] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:20.885416 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:20.886543 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:20.887027 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:20.932395 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 45426.3us
I20251215 08:59:20.882062 21341 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
W20251215 08:59:20.932588 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 46126.6us
I20251215 08:59:20.868708 23193 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:20.888401 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 50419.1us
I20251215 08:59:20.878659 23369 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:20.923842 21337 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:20.886475 23380 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:20.884757 23197 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:20.887241 21378 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:20.887550 23292 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:20.933956 21946 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:20.932770 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:20.991199 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 57384.3us
I20251215 08:59:20.992143 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:20.936630 21379 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:20.923931 21230 leader_election.cc:336] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.555s (SENT)
I20251215 08:59:20.887694 23004 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251215 08:59:20.999898 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.925180 23098 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:20.951808 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:20.926779 23178 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:20.925797 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:20.995251 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 71353.5us
W20251215 08:59:21.004644 21230 leader_election.cc:336] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.557s (SENT)
I20251215 08:59:20.940053 23369 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:20.990587 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.007707 21230 leader_election.cc:336] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.698s (SENT)
W20251215 08:59:21.008417 21230 leader_election.cc:336] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.543s (SENT)
I20251215 08:59:21.009989 21230 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:21.012073 21407 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
W20251215 08:59:20.996362 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 59851.7us
W20251215 08:59:21.001354 21232 leader_election.cc:336] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.555s (SENT)
I20251215 08:59:20.889603 23228 consensus_queue.cc:1059] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20251215 08:59:20.939750 23195 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:20.937892 23243 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:20.937752 23209 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:21.013507 21232 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:21.024279 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:21.030376 21337 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:21.030804 21232 leader_election.cc:336] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.557s (SENT)
W20251215 08:59:21.031277 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 93168.9us
I20251215 08:59:21.031324 21232 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:21.013314 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.032968 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.014722 21379 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:20.998608 23129 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:20.990054 23283 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:21.017982 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.018868 23351 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:21.003070 23027 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:21.001883 23131 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:21.002161 23004 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:21.015802 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:20.942044 23060 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.003566 23098 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:21.067577 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:21.068913 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 181738us
W20251215 08:59:21.070093 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 55482.8us
I20251215 08:59:21.022372 23312 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:21.029620 23375 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:21.114822 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:21.115440 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.116043 23131 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.080914 23210 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:21.118127 21266 raft_consensus.cc:1240] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
W20251215 08:59:21.118671 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 115844us
I20251215 08:59:21.122928 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:21.154803 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:21.086143 22742 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:21.155665 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.036147 23195 raft_consensus.cc:697] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:21.156033 22742 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:21.068027 23190 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:21.156992 23195 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.158048 22742 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:21.159689 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 375) took 2027 ms. Trace:
I20251215 08:59:21.069123 23254 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:21.155121 23378 log.cc:927] Time spent T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
I20251215 08:59:21.127391 23098 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:21.160050 21980 rpcz_store.cc:276] 1215 08:59:19.132310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.239007 (+1106697us) service_pool.cc:225] Handling call
1215 08:59:20.250004 (+ 10997us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:20.250470 (+ 466us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:20.586991 (+336521us) log.cc:844] Serialized 14 byte log entry
1215 08:59:20.887135 (+300144us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:20.925716 (+ 38581us) log.cc:844] Serialized 52 byte log entry
1215 08:59:20.925845 (+ 129us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:20.925882 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.925941 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:21.159458 (+233517us) raft_consensus.cc:1685] finished
1215 08:59:21.159476 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:21.159619 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":275,"raft.queue_time_us":1135,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":36917,"thread_start_us":235297,"threads_started":1,"wal-append.queue_time_us":389023}
W20251215 08:59:21.144814 21232 leader_election.cc:336] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.698s (SENT)
I20251215 08:59:21.161899 21232 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:21.163573 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1567) took 1246 ms. Trace:
W20251215 08:59:21.164534 23212 consensus_meta.cc:287] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.602s user 0.011s sys 0.000s
W20251215 08:59:21.132511 23372 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.255s user 0.000s sys 0.001s
I20251215 08:59:21.087682 23231 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:21.081177 22925 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:21.176403 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 178) took 1885 ms. Trace:
I20251215 08:59:21.163934 21332 rpcz_store.cc:276] 1215 08:59:19.917150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.465101 (+547951us) service_pool.cc:225] Handling call
1215 08:59:20.555459 (+ 90358us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:20.555773 (+ 314us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.555811 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:20.560135 (+ 4324us) log.cc:844] Serialized 26 byte log entry
1215 08:59:20.797491 (+237356us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:20.797546 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.797654 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:21.163356 (+365702us) raft_consensus.cc:1685] finished
1215 08:59:21.163386 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:21.163514 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":178073,"threads_started":1,"wal-append.queue_time_us":278375}
I20251215 08:59:21.118350 23004 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:21.086375 23027 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:21.176963 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1525) took 3907 ms. Trace:
I20251215 08:59:21.178383 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1590) took 1564 ms. Trace:
I20251215 08:59:21.187831 21414 rpcz_store.cc:276] 1215 08:59:17.269386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.456294 (+186908us) service_pool.cc:225] Handling call
1215 08:59:20.375399 (+2919105us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.376238 (+ 839us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:21.176764 (+800526us) spinlock_profiling.cc:243] Waited 2.92 s on lock 0x7b540034f650. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:21.176902 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6712714112}
I20251215 08:59:21.189365 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.190069 21414 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:21.190886 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.186786 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:21.176834 21340 rpcz_store.cc:276] 1215 08:59:19.291116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.291219 (+ 103us) service_pool.cc:225] Handling call
1215 08:59:19.291465 (+ 246us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:19.318586 (+ 27121us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:21.176159 (+1857573us) spinlock_profiling.cc:243] Waited 5.89 ms on lock 0x7b540009bfd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:21.176331 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13538560}
I20251215 08:59:21.194905 23027 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:21.190806 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.198720 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1147) took 1229 ms. Trace:
I20251215 08:59:21.199175 21340 rpcz_store.cc:276] 1215 08:59:19.969426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.197266 (+1227840us) service_pool.cc:225] Handling call
1215 08:59:21.198648 (+ 1382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:21.070434 23243 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.160674 23014 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:21.202762 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:21.188328 21259 rpcz_store.cc:276] 1215 08:59:19.613828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.892631 (+278803us) service_pool.cc:225] Handling call
1215 08:59:19.907403 (+ 14772us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:19.907640 (+ 237us) raft_consensus.cc:1534] Early marking committed up to index 4
1215 08:59:20.885915 (+978275us) log.cc:844] Serialized 14 byte log entry
1215 08:59:21.177958 (+292043us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:21.178021 (+ 63us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:21.178088 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:21.178181 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:21.178315 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":831014,"prepare.queue_time_us":907250,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":1793,"raft.queue_time_us":141912,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":382847,"threads_started":3,"wal-append.queue_time_us":189693}
I20251215 08:59:21.128935 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:21.193742 23382 log.cc:927] Time spent T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
I20251215 08:59:21.096105 23283 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.188395 23375 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:21.226923 21413 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:21.228660 21407 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
W20251215 08:59:21.229310 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.221410 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1584) took 1618 ms. Trace:
W20251215 08:59:21.234810 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.216028 23263 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:21.262729 21977 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 496) took 3292 ms (client timeout 1543 ms). Trace:
W20251215 08:59:21.264796 23385 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20251215 08:59:21.263178 21977 rpcz_store.cc:269] 1215 08:59:17.970351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:17.972893 (+ 2542us) service_pool.cc:225] Handling call
1215 08:59:21.260179 (+3287286us) spinlock_profiling.cc:243] Waited 2.6 s on lock 0x7b54003ebcd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:21.262649 (+ 2470us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5988527360}
I20251215 08:59:21.268764 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:21.270690 21230 leader_election.cc:336] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.889s (SENT)
W20251215 08:59:21.278249 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.284807 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:21.177811 23212 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:21.313990 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.314527 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.314841 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.345103 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.385028 21260 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.014s sys 0.031s
W20251215 08:59:21.386556 21260 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.600s user 0.015s sys 0.031s
I20251215 08:59:21.508507 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:21.509685 21259 raft_consensus.cc:2410] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251215 08:59:21.513469 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 143019us
W20251215 08:59:21.535236 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 264563us
W20251215 08:59:21.535563 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.536031 21230 leader_election.cc:336] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.894s (SENT)
I20251215 08:59:21.536453 21230 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:21.540174 23375 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251215 08:59:21.547855 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.558136 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.507091 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.306468 23193 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:21.670728 23389 log.cc:927] Time spent T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.409s user 0.000s sys 0.000s
W20251215 08:59:21.678614 23312 consensus_meta.cc:287] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.547s user 0.007s sys 0.000s
I20251215 08:59:21.679116 23312 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:21.620874 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.680024 23131 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:21.691810 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.692340 22895 consensus_meta.cc:287] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.966s user 0.009s sys 0.000s
W20251215 08:59:21.692637 23098 consensus_meta.cc:287] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.531s user 0.001s sys 0.005s
I20251215 08:59:21.692982 23098 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:21.694888 23098 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:21.557883 23392 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:21.698504 23212 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:21.700902 21334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1566) took 1807 ms (client timeout 2111 ms). Trace:
I20251215 08:59:21.701386 21339 raft_consensus.cc:1240] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:21.701608 21414 raft_consensus.cc:2393] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:21.620354 23060 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:21.701438 21334 rpcz_store.cc:269] 1215 08:59:19.893636 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.260260 (+366624us) service_pool.cc:225] Handling call
1215 08:59:21.700620 (+1440360us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b5400123ed4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:21.700816 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62985472}
I20251215 08:59:21.531350 23027 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.710819 23027 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:21.712085 21230 leader_election.cc:336] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.715s (SENT)
W20251215 08:59:21.712713 21230 leader_election.cc:336] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.737s (SENT)
I20251215 08:59:21.715663 23004 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.717849 23004 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:21.720719 23369 consensus_meta.cc:287] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.704s user 0.007s sys 0.002s
I20251215 08:59:21.702426 23243 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:21.721247 23369 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:21.717864 23283 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:21.725073 23369 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:21.725512 23178 consensus_meta.cc:287] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.719s user 0.013s sys 0.001s
W20251215 08:59:21.730157 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.731117 21379 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.041s sys 0.055s
W20251215 08:59:21.732820 21379 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.041s sys 0.055s
W20251215 08:59:21.735236 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 612351us
I20251215 08:59:21.740001 21379 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:21.749544 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:21.692808 22895 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:21.775336 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 376058us
I20251215 08:59:21.778605 21379 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:21.782341 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:21.775031 21305 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.032s sys 0.068s
W20251215 08:59:21.789796 21305 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s user 0.033s sys 0.068s
W20251215 08:59:21.790457 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.790858 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.797662 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.798394 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.798734 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.789120 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:21.805691 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:21.806608 21266 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:21.780169 23387 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:21.799256 23263 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:21.788475 23243 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:21.800819 23178 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:21.811161 22925 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:21.811851 23263 consensus_queue.cc:1059] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251215 08:59:21.813850 23262 consensus_meta.cc:287] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.932s user 0.006s sys 0.008s
I20251215 08:59:21.814064 23263 raft_consensus.cc:3055] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:21.808074 23240 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:21.818852 23387 consensus_queue.cc:1059] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:21.840601 23387 raft_consensus.cc:3055] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:21.821832 21379 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:21.830747 23364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.018s sys 0.062s
W20251215 08:59:21.835753 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 255364us
I20251215 08:59:21.844839 21379 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:21.836205 23300 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:21.836658 21332 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.012s sys 0.010s
I20251215 08:59:21.815121 23263 raft_consensus.cc:740] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:21.838512 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1592) took 1529 ms. Trace:
I20251215 08:59:21.849995 21260 rpcz_store.cc:276] 1215 08:59:20.308849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.309614 (+ 765us) service_pool.cc:225] Handling call
1215 08:59:20.309858 (+ 244us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.310026 (+ 168us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:21.387834 (+1077808us) log.cc:844] Serialized 14 byte log entry
1215 08:59:21.838115 (+450281us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:21.838167 (+ 52us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:21.838234 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:21.838298 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:21.838456 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":431061,"prepare.queue_time_us":506985,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":24126,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":1027949,"threads_started":2}
W20251215 08:59:21.841920 23197 consensus_meta.cc:287] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.840s user 0.008s sys 0.000s
W20251215 08:59:21.848811 21332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s user 0.012s sys 0.010s
W20251215 08:59:21.848626 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 259514us
I20251215 08:59:21.838531 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1580) took 1082 ms. Trace:
W20251215 08:59:21.844125 23364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.972s user 0.018s sys 0.062s
I20251215 08:59:21.853907 21977 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:21.858299 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:21.854650 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:21.848052 23399 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:21.856390 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:21.856647 21946 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:21.859349 21260 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:21.843397 23387 raft_consensus.cc:740] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:21.859944 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:21.859831 23197 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:21.875278 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.856806 21413 rpcz_store.cc:276] 1215 08:59:20.756207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.227701 (+471494us) service_pool.cc:225] Handling call
1215 08:59:21.258337 (+ 30636us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:21.506833 (+248496us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:21.506878 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:21.532032 (+ 25154us) log.cc:844] Serialized 26 byte log entry
1215 08:59:21.835883 (+303851us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:21.835944 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:21.836611 (+ 667us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:21.838021 (+ 1410us) raft_consensus.cc:1685] finished
1215 08:59:21.838380 (+ 359us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:21.838481 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8264,"raft.queue_time_us":149086,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":155,"thread_start_us":174743,"threads_started":1,"wal-append.queue_time_us":275457}
I20251215 08:59:21.869613 21973 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:21.879719 23263 consensus_queue.cc:260] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:21.871184 23387 consensus_queue.cc:260] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:21.874058 22895 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:21.815376 23178 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:21.895249 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:21.971441 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:21.866683 23404 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:21.814291 23262 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:21.903046 23048 consensus_meta.cc:287] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.076s user 0.010s sys 0.001s
W20251215 08:59:21.953224 21232 leader_election.cc:336] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.715s (SENT)
W20251215 08:59:21.965662 21230 leader_election.cc:336] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.716s (SENT)
W20251215 08:59:21.967444 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:21.967726 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 122939us
I20251215 08:59:22.032213 21232 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:22.033248 21230 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:22.033391 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 80236.6us
I20251215 08:59:21.969529 23399 raft_consensus.cc:993] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:21.975687 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1562) took 4142 ms. Trace:
W20251215 08:59:21.994851 21944 leader_election.cc:336] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.524s (SENT)
I20251215 08:59:21.871752 23197 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:22.034212 21232 leader_election.cc:336] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.737s (SENT)
I20251215 08:59:22.048769 21944 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:22.048774 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 83156us
I20251215 08:59:22.048617 21258 rpcz_store.cc:276] 1215 08:59:17.833405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.727434 (+894029us) service_pool.cc:225] Handling call
1215 08:59:20.242429 (+1514995us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.242665 (+ 236us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:20.242705 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:20.242761 (+ 56us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:20.242804 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:20.242876 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:21.975433 (+1732557us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540033a250. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:21.975600 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3483301248}
I20251215 08:59:21.973317 23387 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:22.030941 23294 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:21.972747 23314 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
W20251215 08:59:22.058884 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 64096us
I20251215 08:59:22.050426 21232 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:22.036517 23237 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20251215 08:59:22.031832 23404 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.123754 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.132658 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1485) took 1567 ms. Trace:
I20251215 08:59:22.124890 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.241568 21339 rpcz_store.cc:276] 1215 08:59:20.564945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.857374 (+292429us) service_pool.cc:225] Handling call
1215 08:59:21.701204 (+843830us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:21.701811 (+ 607us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:22.132477 (+430666us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b5400123ed0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:22.132599 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1940027648}
W20251215 08:59:22.142406 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 108648us
I20251215 08:59:22.243090 21379 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:22.155391 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:22.180213 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.243224 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:22.119112 21230 leader_election.cc:336] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.541s (SENT)
I20251215 08:59:22.247956 21230 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:22.249589 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 130517us
W20251215 08:59:22.181484 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 324970us
I20251215 08:59:22.125380 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.125619 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.143869 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.253710 21334 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:22.254678 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:22.255234 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.252605 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.256368 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:22.257179 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:22.257762 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:22.257599 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 223403us
I20251215 08:59:22.263012 21334 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:22.264045 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:22.264477 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.264539 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:22.265023 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:22.265156 21230 leader_election.cc:336] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.049s (SENT)
W20251215 08:59:22.265702 21232 leader_election.cc:336] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 2.219s (SENT)
I20251215 08:59:22.266949 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.050266 23407 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:22.268422 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:22.271239 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.060809 23207 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:22.061668 23060 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:22.274027 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.269831 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.060096 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:22.273208 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.131261 23304 raft_consensus.cc:993] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:22.098186 23322 raft_consensus.cc:1081] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:22.050402 22895 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.207s
I20251215 08:59:22.247226 21339 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:22.279055 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.269667 23190 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:22.249800 23416 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:22.276201 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.106237 23178 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:22.128618 23197 consensus_queue.cc:1059] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.151s
I20251215 08:59:21.229992 21264 rpcz_store.cc:276] 1215 08:59:19.602850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.609127 (+ 6277us) service_pool.cc:225] Handling call
1215 08:59:19.609364 (+ 237us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:19.609533 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:20.708207 (+1098674us) log.cc:844] Serialized 14 byte log entry
1215 08:59:21.215621 (+507414us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:21.215673 (+ 52us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:21.221075 (+ 5402us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:21.221173 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:21.221324 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1100120,"prepare.queue_time_us":1196450,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":2412,"raft.queue_time_us":8429,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":3005,"thread_start_us":498657,"threads_started":1}
I20251215 08:59:22.279879 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.267549 23364 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:22.079290 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.291381 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1571) took 2346 ms. Trace:
I20251215 08:59:22.292817 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1570) took 1539 ms. Trace:
I20251215 08:59:22.313925 21341 rpcz_store.cc:276] 1215 08:59:19.944835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.699468 (+754633us) service_pool.cc:225] Handling call
1215 08:59:20.881885 (+182417us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:20.941254 (+ 59369us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:22.291194 (+1349940us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b540027a1d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:22.291320 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":418855680}
I20251215 08:59:22.314401 21414 rpcz_store.cc:276] 1215 08:59:20.753427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.190670 (+437243us) service_pool.cc:225] Handling call
1215 08:59:22.292625 (+1101955us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b54000eae54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:22.292754 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1161110144}
I20251215 08:59:22.315258 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:22.310626 21339 raft_consensus.cc:2393] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:22.257215 21340 raft_consensus.cc:2368] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:22.049963 23048 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.303392 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:22.288879 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34992 (request call id 21) took 1864 ms. Trace:
W20251215 08:59:22.313066 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.319016 21415 rpcz_store.cc:276] 1215 08:59:20.424822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.424965 (+ 143us) service_pool.cc:225] Handling call
1215 08:59:22.288711 (+1863746us) spinlock_profiling.cc:243] Waited 43.8 ms on lock 0x7b540026ded4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bae4d3 00007f398cb6c13a 00007f3986107934 00007f39861078c2 00007f39861077f1 00007f398610776c 00007f3986106a22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:22.288812 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100668672}
I20251215 08:59:22.302588 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.295152 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1557) took 1581 ms. Trace:
I20251215 08:59:22.317215 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.103025 23262 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:22.302227 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.301486 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:22.296975 21409 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1559) took 1581 ms (client timeout 1640 ms). Trace:
I20251215 08:59:22.326136 21266 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20251215 08:59:22.330204 21409 rpcz_store.cc:269] 1215 08:59:20.715864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.716322 (+ 458us) service_pool.cc:225] Handling call
1215 08:59:22.296812 (+1580490us) spinlock_profiling.cc:243] Waited 6.26 ms on lock 0x7b54000d16d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:22.296914 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14389504}
I20251215 08:59:22.320616 21341 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:22.312002 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:22.331848 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.331729 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:22.314015 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.324160 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:22.320235 21406 rpcz_store.cc:276] 1215 08:59:20.714095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.730305 (+ 16210us) service_pool.cc:225] Handling call
1215 08:59:22.294481 (+1564176us) spinlock_profiling.cc:243] Waited 8.21 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:22.294673 (+ 192us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:22.294855 (+ 182us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:22.294879 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:22.294914 (+ 35us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:22.294945 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:22.294998 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:22.295097 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18885376}
I20251215 08:59:22.332710 21334 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:22.332832 21258 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:22.299584 22895 raft_consensus.cc:3055] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:22.320880 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:22.322221 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.321568 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.323935 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:22.312378 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 60651us
I20251215 08:59:22.335137 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.297075 21973 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:22.338253 21381 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:22.279958 21230 leader_election.cc:336] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.653s (SENT)
I20251215 08:59:22.336608 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:22.339601 21230 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:22.281248 23302 raft_consensus.cc:1081] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:22.298381 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:22.280639 23268 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20251215 08:59:22.333802 23409 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:22.339066 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.281893 23267 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:22.364117 21337 raft_consensus.cc:2393] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:22.340391 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.364805 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.365345 21332 raft_consensus.cc:3055] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:22.365482 21262 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:22.365741 21332 raft_consensus.cc:740] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:22.335780 21264 raft_consensus.cc:2368] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4.
I20251215 08:59:22.341562 21257 raft_consensus.cc:2368] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6.
I20251215 08:59:22.343777 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.367197 21339 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:22.367252 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.368105 21333 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:22.346282 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.347711 21341 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:22.369460 21265 raft_consensus.cc:2410] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:22.350252 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.337427 21340 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251215 08:59:22.366360 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.309801 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:22.371035 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.371336 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.413050 21266 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:59:22.414175 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:22.414855 21266 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:22.414892 21944 leader_election.cc:336] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.630s (SENT)
I20251215 08:59:22.373224 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.373584 23302 consensus_queue.cc:1059] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:22.374362 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.420007 21264 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:22.420859 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.421890 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:22.425446 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.333879 22895 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:22.427614 21264 raft_consensus.cc:2393] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:22.427759 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.428339 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.427968 22895 consensus_queue.cc:260] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.428784 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:22.429137 22895 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:22.438628 21336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 176) took 3148 ms (client timeout 1976 ms). Trace:
W20251215 08:59:22.439038 21336 rpcz_store.cc:269] 1215 08:59:19.290055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:19.290178 (+ 123us) service_pool.cc:225] Handling call
1215 08:59:22.438377 (+3148199us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b54001a9ad4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:22.438534 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1410586880}
I20251215 08:59:22.440246 21338 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:22.385586 23236 raft_consensus.cc:3055] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:22.388576 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:22.388855 23339 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:22.389611 23409 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:22.390745 21332 consensus_queue.cc:260] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.443063 21332 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:22.366561 23419 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:22.460676 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:22.401947 23300 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:22.403694 23180 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:22.486666 21973 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 6
I20251215 08:59:22.486697 23300 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:22.491835 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:22.405112 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.288797 23415 consensus_queue.cc:1059] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.117s
I20251215 08:59:22.291354 23416 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:22.493815 23416 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:22.493873 21412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1563) took 1757 ms (client timeout 2215 ms). Trace:
W20251215 08:59:22.494340 21412 rpcz_store.cc:269] 1215 08:59:20.736183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.736305 (+ 122us) service_pool.cc:225] Handling call
1215 08:59:22.404976 (+1668671us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:22.493810 (+ 88834us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11865216}
I20251215 08:59:22.312117 23251 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:22.406123 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:22.495898 23251 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.407153 22955 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:22.410473 23048 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:22.496935 22955 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.305043 23197 raft_consensus.cc:3055] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:22.498332 23251 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:22.498688 23197 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:22.499624 22955 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:22.426671 21976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 511) took 1425 ms. Trace:
I20251215 08:59:22.429369 21259 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:22.501272 21976 rpcz_store.cc:276] 1215 08:59:21.001121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.003531 (+ 2410us) service_pool.cc:225] Handling call
1215 08:59:22.309640 (+1306109us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:22.426571 (+116931us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26884480}
I20251215 08:59:22.504626 21263 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:22.505223 23181 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:22.429934 21264 raft_consensus.cc:2410] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251215 08:59:22.505561 23416 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.505788 23181 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.506345 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.506925 21264 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:22.507520 23416 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:22.508587 21259 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:22.431305 21262 raft_consensus.cc:1240] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:22.378243 23404 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:22.393260 23254 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:22.466296 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 80293.1us
I20251215 08:59:22.510368 23254 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.511170 21232 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:22.466872 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.512212 23254 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:22.513571 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:22.515347 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:22.515679 23262 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:22.516842 22955 raft_consensus.cc:3055] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:22.517514 23415 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:22.518143 23415 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:22.518501 23415 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:22.463016 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 183272us
I20251215 08:59:22.466353 23409 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:22.466722 23236 raft_consensus.cc:740] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:22.520941 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:22.521548 23236 consensus_queue.cc:260] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.522486 23236 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:22.465961 23407 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:22.516055 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:22.499521 23197 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.548010 23197 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:22.551261 23416 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:22.411121 23366 raft_consensus.cc:3055] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:22.342259 23399 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:22.430902 21266 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:22.552135 23366 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:22.517458 22955 raft_consensus.cc:740] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:22.553097 23366 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.553390 22955 consensus_queue.cc:260] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.555161 22955 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:22.555478 21412 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20251215 08:59:22.555811 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.527312 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:22.496127 21406 raft_consensus.cc:606] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:22.406823 21971 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:22.559513 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 383) took 1558 ms. Trace:
I20251215 08:59:22.559670 21973 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
W20251215 08:59:22.533375 21946 leader_election.cc:336] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.966s (SENT)
W20251215 08:59:22.535392 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.560410 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:22.561933 21409 raft_consensus.cc:1240] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251215 08:59:22.562254 23404 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:22.537972 23409 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:22.539774 21230 leader_election.cc:336] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.219s (SENT)
I20251215 08:59:22.563422 21230 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:22.565502 21262 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:22.565986 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.567306 21381 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:22.568740 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 57688.4us
I20251215 08:59:22.566985 22961 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:22.573489 22961 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.575345 22961 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:22.576122 23420 log.cc:927] Time spent T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.429s user 0.002s sys 0.000s
W20251215 08:59:22.577777 21230 leader_election.cc:336] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.111s (SENT)
W20251215 08:59:22.578564 21230 leader_election.cc:336] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.534s (SENT)
I20251215 08:59:22.578925 21230 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:22.579392 21229 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:22.470925 23290 consensus_queue.cc:1059] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.096s
I20251215 08:59:22.585287 21266 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:22.585822 21266 raft_consensus.cc:606] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:22.585775 21381 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:22.586535 21266 raft_consensus.cc:614] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:22.588590 21232 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:22.590065 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:22.599875 21944 leader_election.cc:336] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.976s (SENT)
I20251215 08:59:22.600553 21944 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:22.604041 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.604758 21409 raft_consensus.cc:2368] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
I20251215 08:59:22.604782 21412 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:22.467989 23339 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.605681 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:22.626647 21946 leader_election.cc:336] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.630s (SENT)
I20251215 08:59:22.627346 21946 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:22.686880 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 108338us
I20251215 08:59:22.582209 23314 raft_consensus.cc:697] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:22.689155 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.584300 23180 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:22.690124 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.690301 21414 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20251215 08:59:22.691195 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 77584.1us
I20251215 08:59:22.693593 21412 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:22.702518 21379 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:22.588101 23048 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:22.706952 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.707731 21412 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:22.603865 23274 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:22.471441 23419 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:22.611765 23404 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:22.718305 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.612833 23268 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:22.707165 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:22.691685 23190 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:22.692171 23290 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:22.745998 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:22.752348 21412 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:22.753578 21334 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2c10c3de775846868774c98cc0a75094"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:22.754142 21334 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:22.754472 21334 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:22.757835 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.758682 21332 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:22.686244 23251 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.051s
I20251215 08:59:22.736608 21414 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:22.685664 23178 raft_consensus.cc:3055] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:22.689533 23303 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:22.690829 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:22.697045 22961 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:22.780495 21337 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:22.781096 21337 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:22.736300 23268 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:22.722143 23254 consensus_queue.cc:1059] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20251215 08:59:22.802146 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:22.559996 21971 rpcz_store.cc:276] 1215 08:59:21.000754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.001771 (+ 1017us) service_pool.cc:225] Handling call
1215 08:59:22.406436 (+1404665us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:22.406640 (+ 204us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:22.559162 (+152522us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:22.559434 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11872512}
I20251215 08:59:22.721954 23363 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:22.717489 23403 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:22.707543 23407 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:22.764849 23327 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:22.805049 23268 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:22.774662 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:22.778487 21944 leader_election.cc:336] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.769s (SENT)
I20251215 08:59:22.791615 21411 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:22.782064 23237 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:22.744189 21232 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:22.821607 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 77529.6us
I20251215 08:59:22.781467 21337 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:22.761613 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:22.798045 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.825332 21332 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:22.799934 21334 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:22.826205 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 70116.3us
I20251215 08:59:22.803103 21333 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
W20251215 08:59:22.777199 21307 connection.cc:232] RPC call timeout handler was delayed by -4.7149825340000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
I20251215 08:59:22.706796 23157 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:22.761276 23274 raft_consensus.cc:697] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:22.760913 23294 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:22.842325 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251215 08:59:22.814356 23403 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:22.844754 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:22.770572 23178 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:22.771463 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.830404 21334 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:22.852245 21409 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:22.831110 21412 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:22.793826 23290 raft_consensus.cc:697] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:22.739250 23387 consensus_meta.cc:287] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.670s user 0.009s sys 0.000s
I20251215 08:59:22.836684 22961 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:22.858011 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.856957 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251215 08:59:22.873720 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 56719.2us
I20251215 08:59:22.884227 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:22.844293 23237 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.851740 23403 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:22.890287 22961 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:22.820154 23407 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.825490 23300 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:22.896564 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:22.897653 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 69369.7us
I20251215 08:59:22.843796 23157 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.906870 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.884145 23350 raft_consensus.cc:3055] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:22.829814 23327 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:22.913430 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.852169 23294 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.916023 21409 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
W20251215 08:59:22.918525 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.927750 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:22.891435 23409 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:22.813488 23314 consensus_queue.cc:237] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.904942 22961 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:22.914601 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.859261 23274 consensus_queue.cc:237] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:22.996176 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.900547 23200 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:22.934681 21337 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:22.976912 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 71267.6us
I20251215 08:59:23.005558 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:23.006417 21260 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251215 08:59:22.869534 21258 consensus_meta.cc:287] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.535s user 0.005s sys 0.005s
I20251215 08:59:22.980651 23157 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:22.918013 23350 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:22.982148 23407 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:22.983031 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:22.986846 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:22.932163 21332 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:23.016898 21409 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:23.007819 21258 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:22.877978 23415 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:22.925981 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.010552 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:23.022291 21260 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:23.011152 21337 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.011996 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:23.025128 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:23.025840 21266 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:23.012166 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:23.026634 21262 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:23.018472 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:23.031657 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:23.018530 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 193034us
I20251215 08:59:23.041106 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:23.042819 21230 leader_election.cc:336] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.176s (SENT)
I20251215 08:59:23.043373 21230 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:22.907872 23178 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.051950 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:23.016104 23157 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:23.055475 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.065503 21412 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:23.066748 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.075134 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:22.973322 23290 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.047863 23407 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
W20251215 08:59:23.091785 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.092557 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.081333 21266 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:23.086315 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.093784 21414 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:23.093892 21409 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:23.089241 21260 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:23.094029 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:23.094779 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.095580 21414 raft_consensus.cc:2410] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251215 08:59:23.095494 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:23.096378 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.096408 21260 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:23.099735 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:23.102196 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.104537 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.074606 23178 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:23.085611 23236 consensus_meta.cc:287] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.563s user 0.010s sys 0.001s
W20251215 08:59:23.108161 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 75905.1us
W20251215 08:59:23.119704 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:23.120100 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.122948 21336 pending_rounds.cc:85] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 1
I20251215 08:59:23.123263 21336 pending_rounds.cc:107] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251215 08:59:23.123519 21336 raft_consensus.cc:2889] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:23.109123 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:23.082973 23429 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:23.144189 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.148159 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:23.152446 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.099493 23430 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:23.157160 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:23.116295 23407 raft_consensus.cc:697] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:23.146131 21415 consensus_meta.cc:287] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.533s user 0.014s sys 0.000s
I20251215 08:59:23.058140 23431 consensus_queue.cc:1059] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20251215 08:59:23.097607 23251 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:23.097416 23249 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251215 08:59:23.118754 23424 log.cc:927] Time spent T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.264s user 0.000s sys 0.001s
I20251215 08:59:23.153224 23300 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:23.086884 23157 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.129721 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.168164 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.166326 23443 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:23.164600 21230 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:23.165431 21415 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
W20251215 08:59:23.172564 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 122457us
W20251215 08:59:23.194854 21307 connection.cc:232] RPC call timeout handler was delayed by -4.0587640890000785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20251215 08:59:23.175823 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:23.175180 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.148036 23429 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:23.084507 23350 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.205037 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.199174 21409 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:23.160243 23237 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:23.200803 21307 connection.cc:232] RPC call timeout handler was delayed by -3.8195937980000281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
I20251215 08:59:23.202191 21260 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:23.200917 23426 log.cc:927] Time spent T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.275s user 0.000s sys 0.001s
I20251215 08:59:23.159986 23221 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.038s
I20251215 08:59:23.212654 23415 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:23.204506 23429 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:23.205129 21407 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59102 (request call id 177) took 2448 ms (client timeout 1966 ms). Trace:
I20251215 08:59:23.204365 23276 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:23.218900 21407 rpcz_store.cc:269] 1215 08:59:20.800478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.032780 (+232302us) service_pool.cc:225] Handling call
1215 08:59:23.204904 (+2172124us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b5400341a54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:23.205024 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":448676736}
I20251215 08:59:23.220152 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.206351 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.221782 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.222316 21409 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:23.224293 23407 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.105216 23315 consensus_queue.cc:1059] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.053s
I20251215 08:59:23.225570 23315 raft_consensus.cc:3055] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:23.225948 23315 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:23.226574 23197 consensus_meta.cc:287] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.678s user 0.007s sys 0.003s
I20251215 08:59:23.226712 23315 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.228003 23315 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:23.228284 23327 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.199065 23438 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:23.230002 23438 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:23.207561 21307 connection.cc:232] RPC call timeout handler was delayed by -3.8033937980001156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1684s for a response
I20251215 08:59:23.175319 23294 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.208890 23436 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:23.232888 21307 connection.cc:232] RPC call timeout handler was delayed by -3.7214937980000968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20251215 08:59:23.233791 21307 connection.cc:232] RPC call timeout handler was delayed by -3.7061937980000721s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17920000000000003s for a response
I20251215 08:59:23.233580 23436 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:23.234143 21307 connection.cc:232] RPC call timeout handler was delayed by -3.6776698190000161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
W20251215 08:59:23.234509 21307 connection.cc:232] RPC call timeout handler was delayed by -3.403169819000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
W20251215 08:59:23.234956 21307 connection.cc:232] RPC call timeout handler was delayed by -3.3392698190000374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
W20251215 08:59:23.235232 21307 connection.cc:232] RPC call timeout handler was delayed by -3.1511698189999606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21309999999999998s for a response
I20251215 08:59:23.169780 23190 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
W20251215 08:59:23.230441 22895 consensus_meta.cc:287] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.763s user 0.009s sys 0.000s
I20251215 08:59:23.248281 23436 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:23.249888 22955 consensus_meta.cc:287] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.694s user 0.010s sys 0.000s
W20251215 08:59:23.251416 21263 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35486 (request call id 152) took 4475 ms (client timeout 3214 ms). Trace:
W20251215 08:59:23.251863 21263 rpcz_store.cc:269] 1215 08:59:18.776536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:18.892650 (+116114us) service_pool.cc:225] Handling call
1215 08:59:23.251222 (+4358572us) spinlock_profiling.cc:243] Waited 3.58 s on lock 0x7b54002b4dd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:23.251321 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8226246400}
W20251215 08:59:23.235541 21307 connection.cc:232] RPC call timeout handler was delayed by -2.7272698189999574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2602s for a response
I20251215 08:59:23.201503 21230 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:23.254168 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 52767.8us
I20251215 08:59:23.254945 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:23.255043 21415 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:23.164064 23430 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.236939 21262 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:23.237771 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.265712 21407 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:23.246572 21378 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:23.266961 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:23.268325 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.234424 23157 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.275137 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:23.275600 21409 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:23.276587 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.277392 21409 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:23.278409 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.279711 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.280582 21409 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:23.270522 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:23.282935 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:23.283286 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.284171 21411 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:23.284981 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.285951 23430 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.232120 23438 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:23.291038 23434 log.cc:927] Time spent T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.130s user 0.001s sys 0.000s
I20251215 08:59:23.274384 23284 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:23.293387 21260 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:23.295193 23268 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.296751 21407 raft_consensus.cc:2368] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:23.286584 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:23.306013 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 59816us
I20251215 08:59:23.312587 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.271932 23299 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:23.207901 23443 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.316895 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.322484 21307 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.308537 23380 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:23.310148 23300 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:23.330336 21307 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.335701 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.317647 23438 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:23.338598 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.356354 21333 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:23.342468 21946 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:23.357873 21946 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.358896 21946 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:23.359648 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.360810 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.346421 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:23.361644 21332 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:23.323836 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:23.326236 23380 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:23.334743 23327 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:23.386366 21332 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
W20251215 08:59:23.354578 23425 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.464s user 0.000s sys 0.001s
I20251215 08:59:23.338585 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:23.357064 21334 raft_consensus.cc:1240] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:23.356909 21232 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:23.388442 21232 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:23.345492 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:23.349416 23268 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:23.370544 21379 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:23.331535 23436 raft_consensus.cc:2764] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251215 08:59:23.325426 23237 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:23.384258 21307 leader_election.cc:400] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
I20251215 08:59:23.389649 21341 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:23.398949 21307 leader_election.cc:403] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:23.404497 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:23.385485 23303 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:23.381851 23209 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:23.331648 23403 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.393311 23299 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:23.387971 23380 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:23.386760 23294 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:23.387120 23325 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:23.387229 23200 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election won for term 4
W20251215 08:59:23.410140 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 71644.1us
I20251215 08:59:23.401013 23190 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:23.411273 23438 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:23.480481 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 63515.9us
I20251215 08:59:23.414413 23370 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:23.416115 23300 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:23.425616 23194 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:23.429668 23299 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:23.431676 23445 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:23.501170 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:23.515683 21264 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:23.398666 23442 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.106s
W20251215 08:59:23.521981 22961 consensus_meta.cc:287] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.523s user 0.009s sys 0.000s
I20251215 08:59:23.522660 21411 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:23.522480 22961 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.523895 23443 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.524727 23299 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:23.528505 23445 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:23.529162 23445 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:23.512805 23300 raft_consensus.cc:697] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:23.505458 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1516) took 1174 ms. Trace:
W20251215 08:59:23.500475 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.535115 21258 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:23.494115 23190 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:23.535709 21260 raft_consensus.cc:1240] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:23.531322 21339 rpcz_store.cc:276] 1215 08:59:22.331422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:22.368050 (+ 36628us) service_pool.cc:225] Handling call
1215 08:59:22.376359 (+ 8309us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:22.376815 (+ 456us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:22.376859 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:22.377451 (+ 592us) log.cc:844] Serialized 26 byte log entry
1215 08:59:22.701587 (+324136us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:22.897904 (+196317us) log.cc:844] Serialized 14 byte log entry
1215 08:59:23.083728 (+185824us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.083804 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:23.354018 (+270214us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:23.505208 (+151190us) raft_consensus.cc:1685] finished
1215 08:59:23.505292 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.505402 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":211697,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":154680,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":6530,"spinlock_wait_cycles":6049664,"thread_start_us":677331,"threads_started":3,"wal-append.queue_time_us":443009}
I20251215 08:59:23.486807 23325 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
W20251215 08:59:23.554895 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.530386 23300 consensus_queue.cc:237] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.512270 23200 raft_consensus.cc:697] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:23.537634 22961 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.546969 23190 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:23.550312 23304 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.234s user 0.017s sys 0.040s
I20251215 08:59:23.536722 23194 raft_consensus.cc:697] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:23.549463 23325 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:23.572077 23304 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.260s user 0.017s sys 0.040s
I20251215 08:59:23.573642 23086 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:23.769197 23194 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:23.641743 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:23.655504 21257 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35486 (request call id 166) took 3254 ms (client timeout 2004 ms). Trace:
W20251215 08:59:23.780051 21257 rpcz_store.cc:269] 1215 08:59:20.400805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.402591 (+ 1786us) service_pool.cc:225] Handling call
1215 08:59:23.654951 (+3252360us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54002422d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:23.655253 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4054802688}
W20251215 08:59:23.659999 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:23.785269 22938 consensus_queue.cc:1059] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:23.785501 23200 consensus_queue.cc:237] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.660209 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1619) took 1278 ms. Trace:
I20251215 08:59:23.661868 21265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1613) took 1281 ms. Trace:
I20251215 08:59:23.710968 21164 catalog_manager.cc:5654] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 2 to 4, leader changed from f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 4 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:23.663092 21266 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 6. Ops: []
I20251215 08:59:23.664577 21261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1605) took 2587 ms. Trace:
W20251215 08:59:23.671241 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:23.674232 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 269697us
I20251215 08:59:23.788491 21261 rpcz_store.cc:276] 1215 08:59:21.077238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.077516 (+ 278us) service_pool.cc:225] Handling call
1215 08:59:21.077914 (+ 398us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:21.159182 (+ 81268us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:21.159225 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:21.159281 (+ 56us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:21.159316 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.663996 (+2504680us) spinlock_profiling.cc:243] Waited 81 ms on lock 0x7b5400323554. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:23.664101 (+ 105us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.664347 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186410240}
I20251215 08:59:23.791029 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:23.794546 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:23.795553 21258 raft_consensus.cc:1240] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:23.678310 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1505) took 2889 ms. Trace:
I20251215 08:59:23.796526 21338 rpcz_store.cc:276] 1215 08:59:20.788681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.166704 (+378023us) service_pool.cc:225] Handling call
1215 08:59:22.440024 (+1273320us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:22.468556 (+ 28532us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.677749 (+1209193us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001a9ad0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:23.678060 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2825277824}
I20251215 08:59:23.692147 21341 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
W20251215 08:59:23.719846 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:23.720700 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 166542us
I20251215 08:59:23.722368 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 180) took 1029 ms. Trace:
I20251215 08:59:23.724157 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1587) took 1033 ms. Trace:
I20251215 08:59:23.725883 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:23.730072 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1379) took 1035 ms. Trace:
I20251215 08:59:23.733872 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:23.738816 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 299881us
I20251215 08:59:23.744822 21264 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:23.763492 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:23.766152 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 384) took 2611 ms. Trace:
I20251215 08:59:23.597280 23299 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:23.817627 21230 leader_election.cc:336] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.677s (SENT)
I20251215 08:59:23.817955 23299 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.818310 21230 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:23.589870 23303 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:23.819600 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1586) took 1130 ms. Trace:
I20251215 08:59:23.820086 21413 rpcz_store.cc:276] 1215 08:59:22.689842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:22.689954 (+ 112us) service_pool.cc:225] Handling call
1215 08:59:23.725436 (+1035482us) spinlock_profiling.cc:243] Waited 8.74 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:23.819488 (+ 94052us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20094208}
I20251215 08:59:23.585255 23307 raft_consensus.cc:3055] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:23.822420 23307 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:23.823807 23307 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.824398 23181 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.595077 23403 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:23.818004 21335 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:23.610390 23419 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.554670 23429 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.597246 23438 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
I20251215 08:59:23.610605 23445 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:23.611661 23451 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.219s user 0.000s sys 0.002s
I20251215 08:59:23.595126 23327 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:23.845474 23429 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.776087 23380 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.848078 23380 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:23.848477 23284 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:23.849426 23284 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.852418 23284 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.854156 23251 raft_consensus.cc:3055] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:23.854637 23251 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:23.855787 23251 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.861593 23251 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:23.861853 23307 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:23.788079 21265 rpcz_store.cc:276] 1215 08:59:22.380437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:22.442040 (+ 61603us) service_pool.cc:225] Handling call
1215 08:59:22.460580 (+ 18540us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:22.539871 (+ 79291us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:22.539919 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:22.539993 (+ 74us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:22.540058 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.661272 (+1121214us) spinlock_profiling.cc:243] Waited 79 ms on lock 0x7b54001342d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:23.661345 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.661664 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181697408}
I20251215 08:59:23.787640 21259 rpcz_store.cc:276] 1215 08:59:22.382086 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:22.510471 (+128385us) service_pool.cc:225] Handling call
1215 08:59:22.510882 (+ 411us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:22.525726 (+ 14844us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:22.525754 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:22.525792 (+ 38us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:22.525823 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.659563 (+1133740us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5400339354. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:23.659645 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.659994 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33668352}
I20251215 08:59:23.799229 21410 rpcz_store.cc:276] 1215 08:59:22.690245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:22.690358 (+ 113us) service_pool.cc:225] Handling call
1215 08:59:23.205563 (+515205us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:23.205744 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:23.205766 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:23.205806 (+ 40us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:23.205844 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.205896 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.723726 (+517830us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400341a50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:23.723928 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1172090112}
I20251215 08:59:23.825634 21413 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251215 08:59:23.836331 23086 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:23.836724 21263 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:23.820456 23303 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.845546 23445 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.841475 21164 catalog_manager.cc:5654] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:23.778549 23189 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:23.869715 23321 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:23.870476 23321 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.798863 21409 rpcz_store.cc:276] 1215 08:59:22.692615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:23.288959 (+596344us) service_pool.cc:225] Handling call
1215 08:59:23.289280 (+ 321us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:23.326582 (+ 37302us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:23.326613 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:23.333668 (+ 7055us) log.cc:844] Serialized 26 byte log entry
1215 08:59:23.467880 (+134212us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:23.467942 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.468043 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:23.721768 (+253725us) raft_consensus.cc:1685] finished
1215 08:59:23.721821 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.722081 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55597,"raft.queue_time_us":36664,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3749,"thread_start_us":113805,"threads_started":1,"wal-append.queue_time_us":211737}
I20251215 08:59:23.873064 21409 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:23.873497 23189 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.876052 23189 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:23.878614 21944 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.600s (SENT)
I20251215 08:59:23.808712 21978 rpcz_store.cc:276] 1215 08:59:21.154682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.156908 (+ 2226us) service_pool.cc:225] Handling call
1215 08:59:21.167666 (+ 10758us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:21.214540 (+ 46874us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:21.214565 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:21.215099 (+ 534us) log.cc:844] Serialized 26 byte log entry
1215 08:59:22.304679 (+1089580us) spinlock_profiling.cc:243] Waited 883 us on lock 0x7b180005c6e0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:22.708100 (+403421us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:22.708141 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.297964 (+589823us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b180005c6e0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b06112 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:23.765490 (+467526us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b54003f02d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:23.765554 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:23.765602 (+ 48us) raft_consensus.cc:1685] finished
1215 08:59:23.765619 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.765880 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9294,"raft.queue_time_us":5975,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":12829,"spinlock_wait_cycles":171025792,"thread_start_us":403237,"threads_started":1,"wal-append.queue_time_us":1597929}
I20251215 08:59:23.867161 23181 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:23.805725 21411 rpcz_store.cc:276] 1215 08:59:22.694835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:23.728073 (+1033238us) service_pool.cc:225] Handling call
1215 08:59:23.728838 (+ 765us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:23.729399 (+ 561us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:23.729446 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:23.729537 (+ 91us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:23.729599 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:23.729705 (+ 106us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:23.729847 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:23.895591 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:23.896577 21260 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:23.897228 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 106245us
I20251215 08:59:23.889174 23314 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:23.884413 23243 consensus_queue.cc:1059] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20251215 08:59:23.886786 21164 catalog_manager.cc:5654] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:23.915841 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:23.916877 23086 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:23.917505 23086 raft_consensus.cc:697] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:23.919492 23339 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.918918 23086 consensus_queue.cc:237] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.921409 23366 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:23.921998 21381 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:23.928238 21414 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:23.949091 21260 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20251215 08:59:23.953981 23176 consensus_queue.cc:1059] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20251215 08:59:23.954962 23301 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:23.950667 21260 pending_rounds.cc:85] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 3
I20251215 08:59:23.959583 21260 pending_rounds.cc:107] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20251215 08:59:23.966698 21260 raft_consensus.cc:2889] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:23.966734 23008 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:23.969066 23008 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.971313 23008 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:23.972776 23181 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:23.973472 23181 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.973886 21407 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:23.974076 23176 raft_consensus.cc:3055] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:23.974687 21974 raft_consensus.cc:1240] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:23.975281 23301 raft_consensus.cc:697] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:23.978219 23261 consensus_queue.cc:1059] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:23.976824 23301 consensus_queue.cc:237] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.945127 23314 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.979365 23261 raft_consensus.cc:3055] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:23.979781 23261 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:23.980779 23314 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:23.974660 23176 raft_consensus.cc:740] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:23.982347 23176 consensus_queue.cc:260] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:23.984086 23176 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:23.985685 21230 leader_election.cc:336] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.035s (SENT)
I20251215 08:59:23.986351 21230 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:23.988171 23189 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:23.980701 23261 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:23.991357 23350 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:23.990530 21164 catalog_manager.cc:5654] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
W20251215 08:59:23.992200 23267 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.209s user 0.015s sys 0.002s
W20251215 08:59:23.992616 23267 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.295s user 0.015s sys 0.003s
I20251215 08:59:23.978339 21379 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:23.993993 23146 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:23.993265 21164 catalog_manager.cc:5654] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:23.994539 23146 raft_consensus.cc:697] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:23.995647 23146 consensus_queue.cc:237] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:23.998461 23263 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:24.001266 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:24.007011 23304 raft_consensus.cc:993] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:24.007709 23304 raft_consensus.cc:1081] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:24.009869 23363 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:24.010411 23363 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:24.013523 23322 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:23.995949 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:23.995755 21164 catalog_manager.cc:5654] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:23.975559 23181 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:24.037720 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.040277 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:24.040971 21260 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:24.046716 21230 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.168s (SENT)
I20251215 08:59:24.050542 21230 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:24.061981 21381 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:24.049815 21972 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:24.061549 21257 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:24.051949 23200 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:24.080013 23325 consensus_meta.cc:287] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.502s user 0.004s sys 0.008s
I20251215 08:59:24.081043 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:24.089270 21341 raft_consensus.cc:1240] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:24.089686 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:24.089609 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:24.093462 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 61491.3us
I20251215 08:59:24.099728 21974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 525) took 1185 ms. Trace:
I20251215 08:59:24.099607 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:24.100720 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:24.101199 23249 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:24.101665 23423 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:24.103256 21972 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251215 08:59:24.107288 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:24.132889 21976 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:24.062280 23181 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:24.109200 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 182996us
I20251215 08:59:24.110262 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 184) took 1416 ms. Trace:
W20251215 08:59:24.103615 21335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1575) took 4157 ms (client timeout 2715 ms). Trace:
I20251215 08:59:24.071527 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:24.129524 21414 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:24.132112 21977 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:24.064532 21164 catalog_manager.cc:5654] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:24.137388 21410 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:24.232235 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 156416us
W20251215 08:59:24.233860 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:24.231573 21335 rpcz_store.cc:269] 1215 08:59:19.946116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.331727 (+385611us) service_pool.cc:225] Handling call
1215 08:59:24.103295 (+3771568us) spinlock_profiling.cc:243] Waited 3.47 s on lock 0x7b5400124dd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:24.103491 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7980992384}
W20251215 08:59:24.189216 23190 consensus_meta.cc:287] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.557s user 0.011s sys 0.001s
I20251215 08:59:24.230561 21407 rpcz_store.cc:276] 1215 08:59:22.693669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:23.723593 (+1029924us) service_pool.cc:225] Handling call
1215 08:59:23.724301 (+ 708us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:23.974287 (+249986us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:24.109947 (+135660us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54002f5dd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:24.110182 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":502276864}
W20251215 08:59:24.242134 21230 leader_election.cc:336] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.215s (SENT)
I20251215 08:59:24.242486 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:24.136299 21975 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:24.131328 23423 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:24.249158 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.249207 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:24.250962 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:24.252022 21307 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:24.253140 23129 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:24.253341 23449 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:24.253803 23129 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.254148 23449 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.255728 21378 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:24.255937 23129 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:24.257311 21408 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:24.259204 23304 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:24.261358 23181 consensus_queue.cc:1059] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:24.262239 23181 raft_consensus.cc:3055] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:24.262591 23181 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:24.238832 21164 catalog_manager.cc:5654] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 2, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:24.263489 23181 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:24.266594 23267 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:24.266814 21406 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:24.267351 23263 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:24.267956 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:24.267879 23263 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.268656 21341 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:24.269065 23181 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:24.272367 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1599) took 1243 ms. Trace:
I20251215 08:59:24.272816 21409 rpcz_store.cc:276] 1215 08:59:23.029189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:23.874045 (+844856us) service_pool.cc:225] Handling call
1215 08:59:23.874435 (+ 390us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:23.883677 (+ 9242us) raft_consensus.cc:1534] Early marking committed up to index 2
1215 08:59:23.944562 (+ 60885us) log.cc:844] Serialized 14 byte log entry
1215 08:59:24.050379 (+105817us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:24.065409 (+ 15030us) log.cc:844] Serialized 52 byte log entry
1215 08:59:24.065611 (+ 202us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:24.065657 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:24.065744 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:24.272120 (+206376us) raft_consensus.cc:1685] finished
1215 08:59:24.272145 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:24.272293 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5305,"prepare.queue_time_us":63932,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":54,"raft.queue_time_us":10227,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":6856,"thread_start_us":140094,"threads_started":2,"wal-append.queue_time_us":189961}
I20251215 08:59:24.274994 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:24.275657 21409 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:24.276484 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:24.279554 23157 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:24.279772 21409 raft_consensus.cc:1240] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:24.286026 23363 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.279779 23086 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251215 08:59:24.280194 23157 raft_consensus.cc:697] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:24.237550 23249 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.288259 21409 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:24.288841 21334 raft_consensus.cc:3055] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:24.253844 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:24.284730 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.293964 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:24.268849 23267 raft_consensus.cc:3055] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:24.297629 21334 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:24.300117 23267 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:24.264986 23423 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:24.299256 21973 raft_consensus.cc:2393] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 2: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:24.190392 23200 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.168222 23325 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.118438 21974 rpcz_store.cc:276] 1215 08:59:22.914763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:22.918967 (+ 4204us) service_pool.cc:225] Handling call
1215 08:59:23.766978 (+848011us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:23.767421 (+ 443us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:23.975282 (+207861us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:24.099434 (+124152us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54003ed854. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:24.099635 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":453342336}
I20251215 08:59:24.318804 21266 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:24.317857 23283 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:24.322512 21339 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:24.324420 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 86394.4us
I20251215 08:59:24.248121 23190 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.329777 21414 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20251215 08:59:24.328616 21944 leader_election.cc:336] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.105s (SENT)
I20251215 08:59:24.327755 21334 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.334230 21334 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:24.335100 23249 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:24.339141 21972 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:24.340167 21229 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:24.341272 23283 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:24.341610 23283 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:24.341717 21229 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:24.341940 23283 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:24.342849 23375 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251215 08:59:24.344537 23375 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:24.345134 23157 consensus_queue.cc:237] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.345631 23284 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:24.347114 23284 consensus_queue.cc:1059] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:24.348037 23363 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:24.348405 21336 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:24.305974 23146 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20251215 08:59:24.352334 21944 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:24.344885 23375 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:24.358505 21408 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1565) took 3619 ms (client timeout 1555 ms). Trace:
I20251215 08:59:24.358282 22799 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20251215 08:59:24.359117 23267 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.360536 23267 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:24.327495 21164 catalog_manager.cc:5654] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:24.366675 21339 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
W20251215 08:59:24.370111 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:24.358954 21408 rpcz_store.cc:269] 1215 08:59:20.751854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:20.771731 (+ 19877us) service_pool.cc:225] Handling call
1215 08:59:22.301329 (+1529598us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:24.358277 (+2056948us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001752d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:24.358404 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4439163136}
W20251215 08:59:24.351310 23478 log.cc:927] Time spent T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20251215 08:59:24.348027 23284 consensus_queue.cc:1243] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer f0824fa5450345719cac22fc6ae3ce2e log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251215 08:59:24.380908 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:24.367857 23375 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:24.384764 21341 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:24.370544 21411 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:24.400861 23436 consensus_queue.cc:1059] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
W20251215 08:59:24.403298 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 86503.3us
I20251215 08:59:24.391070 23129 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:24.405717 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 56104.6us
I20251215 08:59:24.409633 21334 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:24.419562 21263 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:24.426703 21307 leader_election.cc:400] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:24.427075 21307 leader_election.cc:403] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:24.410074 21164 catalog_manager.cc:5654] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:24.445375 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:24.441229 23436 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:24.434151 23157 raft_consensus.cc:3055] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:24.456022 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:24.469444 21259 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:24.459792 21164 catalog_manager.cc:5654] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 2 to 4, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 4 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:24.480854 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:24.481868 21406 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1374) took 1917 ms (client timeout 1668 ms). Trace:
I20251215 08:59:24.482543 23263 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:24.483078 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:24.483374 23249 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:24.483608 21262 raft_consensus.cc:3055] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:24.483731 21264 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:24.484146 23261 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:24.475396 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:24.483114 21415 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:24.485615 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1398) took 1501 ms. Trace:
I20251215 08:59:24.485960 21415 rpcz_store.cc:276] 1215 08:59:22.983832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:23.735533 (+751701us) service_pool.cc:225] Handling call
1215 08:59:24.482659 (+747126us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:24.484844 (+ 2185us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:24.485431 (+ 587us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54002940d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:24.485556 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1717062528}
I20251215 08:59:24.484046 21262 raft_consensus.cc:740] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:24.492796 21258 raft_consensus.cc:2393] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:24.482271 21406 rpcz_store.cc:269] 1215 08:59:22.564618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:22.565737 (+ 1119us) service_pool.cc:225] Handling call
1215 08:59:24.481627 (+1915890us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54002940d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:24.481788 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3857662208}
I20251215 08:59:24.464952 23157 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:24.458500 23325 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:24.503242 21257 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:24.480104 23200 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:24.477837 23190 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:24.509393 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:24.511668 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:24.514043 21334 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:24.514986 21164 catalog_manager.cc:5654] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 2, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 2 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:24.505288 21262 consensus_queue.cc:260] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:24.540714 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.542049 23231 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:24.542570 23231 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:24.542939 23231 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:24.542755 23339 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:24.545028 23173 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20251215 08:59:24.548611 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:24.507694 23283 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.514156 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:24.513552 23303 consensus_queue.cc:1059] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20251215 08:59:24.568538 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:24.540217 21164 catalog_manager.cc:5654] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:24.570441 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:24.568056 23190 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:24.564275 23327 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:24.582635 21415 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:24.592283 21164 catalog_manager.cc:5654] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
W20251215 08:59:24.594067 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.594940 21262 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:24.597409 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.593719 21164 catalog_manager.cc:5654] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:24.598732 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:24.602880 21944 leader_election.cc:336] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.855s (SENT)
I20251215 08:59:24.603410 21944 leader_election.cc:304] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:24.605655 21264 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:24.589869 23303 consensus_queue.cc:1243] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer 195a56ac72a84018880bfb3bf6e6ef1d log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251215 08:59:24.582883 23190 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
W20251215 08:59:24.626102 21944 leader_election.cc:336] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.004s (SENT)
I20251215 08:59:24.626672 21944 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:24.627862 23321 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:24.604580 23363 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:24.609939 23325 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:24.619167 21164 catalog_manager.cc:5654] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:24.580240 23157 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.596753 21340 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:24.641615 23150 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
W20251215 08:59:24.644290 23482 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20251215 08:59:24.614796 23327 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.642223 23325 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:24.647815 21164 catalog_manager.cc:5654] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 1 to 3, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:24.651409 21340 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:24.608052 23263 consensus_queue.cc:1059] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20251215 08:59:24.673815 21340 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:24.673820 21262 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:24.664664 23325 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:24.651213 23173 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20251215 08:59:24.684084 21379 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:24.682783 21971 raft_consensus.cc:3055] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:24.688081 21971 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:24.698979 21971 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.700249 21971 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:24.705682 21412 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:24.705962 21410 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:24.700795 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:24.683184 23303 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:24.683365 23283 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:24.703248 21164 catalog_manager.cc:5654] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 2 to 3, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:24.718446 21415 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:24.700263 21972 raft_consensus.cc:1240] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:24.722261 21263 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:24.703526 23157 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:24.680649 23363 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
W20251215 08:59:24.733351 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 49379.6us
I20251215 08:59:24.719697 21976 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:24.724009 23366 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
W20251215 08:59:24.745399 23484 log.cc:927] Time spent T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20251215 08:59:24.736032 21164 catalog_manager.cc:5654] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 2 to 3, leader changed from f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:24.765384 23477 log.cc:927] Time spent T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.342s user 0.005s sys 0.000s
I20251215 08:59:24.728709 23263 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:24.778573 21944 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.168s (SENT)
I20251215 08:59:24.778949 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 187) took 1409 ms. Trace:
I20251215 08:59:24.781154 21413 rpcz_store.cc:276] 1215 08:59:23.369396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:23.877285 (+507889us) service_pool.cc:225] Handling call
1215 08:59:23.877833 (+ 548us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:23.878532 (+ 699us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:23.878566 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:23.879231 (+ 665us) log.cc:844] Serialized 26 byte log entry
1215 08:59:24.070855 (+191624us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:24.138207 (+ 67352us) log.cc:844] Serialized 14 byte log entry
1215 08:59:24.250977 (+112770us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:24.251053 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:24.764745 (+513692us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:24.770151 (+ 5406us) raft_consensus.cc:1685] finished
1215 08:59:24.778689 (+ 8538us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:24.778875 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112470,"prepare.queue_time_us":97458,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":112806,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":4168064,"thread_start_us":190593,"threads_started":1,"wal-append.queue_time_us":509915}
I20251215 08:59:24.786154 21340 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.774834 23366 raft_consensus.cc:697] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:24.802930 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 276610us
I20251215 08:59:24.812240 21259 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:24.813723 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:24.815848 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:24.815418 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:24.785812 23263 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.824383 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:24.822479 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:24.815333 22925 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20251215 08:59:24.822746 23302 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20251215 08:59:24.811503 23350 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:24.826542 21413 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:24.814338 23449 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:24.829851 21260 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
W20251215 08:59:24.835536 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.838923 23150 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:24.809628 23321 consensus_queue.cc:1059] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20251215 08:59:24.842423 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:24.839624 21381 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:24.825953 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:24.828554 21340 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:24.848626 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:24.844120 21232 leader_election.cc:336] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.602s (SENT)
I20251215 08:59:24.854146 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:24.855719 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:24.863253 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:24.839389 23014 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:24.846737 23339 raft_consensus.cc:697] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:24.882455 23339 consensus_queue.cc:237] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.836545 23327 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:24.855794 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 50417.1us
I20251215 08:59:24.850769 23449 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:24.836249 23231 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.899863 21413 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:24.844651 23350 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.924737 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:24.846592 23150 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:24.928655 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.930172 21336 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:24.882455 23486 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:24.898929 23445 raft_consensus.cc:3055] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:24.909979 23448 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:24.907346 23449 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:24.864296 23375 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.942795 21415 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:24.946949 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:24.924973 23249 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:24.916455 23327 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:24.853420 23366 consensus_queue.cc:237] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:24.899109 23014 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:24.947240 23449 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:24.959401 21339 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:24.960502 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:24.935540 23445 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:24.968154 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:24.963783 22927 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:24.969696 21334 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
W20251215 08:59:24.935565 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 249612us
W20251215 08:59:24.973552 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:24.981642 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:24.966522 23363 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:24.980600 21230 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.135s (SENT) [suppressed 51 similar messages]
I20251215 08:59:24.966478 23486 raft_consensus.cc:697] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:24.998281 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 161422us
I20251215 08:59:24.999851 21232 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:24.949550 23448 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.009488 21232 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:24.965055 23327 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:24.995447 23423 consensus_meta.cc:287] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.667s user 0.006s sys 0.006s
W20251215 08:59:25.016384 21230 leader_election.cc:336] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.715s (SENT)
I20251215 08:59:24.992871 23200 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:24.982918 23321 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20251215 08:59:24.997032 23263 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:25.026871 23375 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:25.029667 23284 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:25.030184 21230 leader_election.cc:336] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.803s (SENT)
W20251215 08:59:25.033063 23490 log.cc:927] Time spent T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20251215 08:59:25.019536 23249 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:25.073802 23284 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.074642 23249 raft_consensus.cc:697] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:25.073813 21230 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:25.036388 21413 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:25.075939 23190 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:25.075794 23249 consensus_queue.cc:237] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:25.037617 23492 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20251215 08:59:25.075937 23284 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:25.076707 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 85049.9us
W20251215 08:59:25.046953 23491 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20251215 08:59:25.036370 23486 consensus_queue.cc:237] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:25.012275 23407 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20251215 08:59:25.049774 23150 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:25.051308 23014 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:25.052783 23231 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.015779 23181 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:24.986665 22927 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:25.076179 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 46040.9us
I20251215 08:59:25.049067 23448 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:25.130590 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 51697.7us
I20251215 08:59:25.079386 23339 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:25.133745 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:25.136549 23267 consensus_meta.cc:287] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.776s user 0.012s sys 0.001s
I20251215 08:59:25.020287 23423 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:25.174091 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 192507us
I20251215 08:59:25.176815 21946 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:25.199723 21407 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 2.
I20251215 08:59:25.021402 23445 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.172942 23181 raft_consensus.cc:697] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:25.176728 23350 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:25.210561 21230 leader_election.cc:336] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.065s (SENT)
I20251215 08:59:25.220192 21230 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:25.220477 22927 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:25.199952 23339 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.225292 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:25.226678 21971 consensus_meta.cc:287] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.526s user 0.008s sys 0.004s
W20251215 08:59:25.226908 23493 log.cc:927] Time spent T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20251215 08:59:25.227456 21971 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251215 08:59:25.227612 23339 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.192641 23267 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election won for term 5
W20251215 08:59:25.220072 21412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1399) took 2235 ms (client timeout 1662 ms). Trace:
W20251215 08:59:25.232275 21412 rpcz_store.cc:269] 1215 08:59:22.984133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:23.067859 (+ 83726us) service_pool.cc:225] Handling call
1215 08:59:25.219795 (+2151936us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54002945d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:25.219971 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3021521280}
I20251215 08:59:25.237160 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:25.220077 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:25.212232 23086 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20251215 08:59:25.238771 23284 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:25.269570 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:25.269780 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 84669us
I20251215 08:59:25.248571 23445 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:25.275101 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:25.277076 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:25.282171 21408 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:25.287849 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:25.262074 23339 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:25.289808 21340 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:25.284235 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:25.298646 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:25.299239 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:25.300315 23497 log.cc:927] Time spent T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20251215 08:59:25.317795 23232 raft_consensus.cc:3055] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:25.318477 23232 raft_consensus.cc:740] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:25.319610 23232 consensus_queue.cc:260] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.336138 21408 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:25.338672 21414 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:25.342094 21341 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:25.344074 23267 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:25.344607 23284 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:25.344851 23423 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:25.338691 23129 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20251215 08:59:25.345252 23284 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.326743 23327 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.346061 23502 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:25.346657 23502 raft_consensus.cc:697] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:25.347421 23284 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.347573 23502 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.295161 23339 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:25.374344 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:25.361055 21976 consensus_meta.cc:287] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.600s user 0.005s sys 0.005s
I20251215 08:59:25.359812 23339 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:25.390033 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 95619us
I20251215 08:59:25.390799 21946 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:25.394723 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:25.396809 23474 log.cc:927] Time spent T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251215 08:59:25.356398 23449 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.341773 23181 consensus_queue.cc:237] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.410770 23515 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
W20251215 08:59:25.412539 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:25.402868 23504 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.013s
I20251215 08:59:25.392777 21976 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:25.420603 21307 leader_election.cc:400] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4.
I20251215 08:59:25.423398 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:25.423987 21971 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:25.424256 23157 consensus_meta.cc:287] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.674s user 0.002s sys 0.006s
I20251215 08:59:25.360088 23267 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.420945 21307 leader_election.cc:403] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251215 08:59:25.439002 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:25.408008 23283 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:25.433792 23325 consensus_meta.cc:287] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.740s user 0.001s sys 0.010s
I20251215 08:59:25.431610 23515 raft_consensus.cc:697] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
W20251215 08:59:25.442785 23503 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20251215 08:59:25.443096 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:25.437213 23157 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4.
W20251215 08:59:25.448292 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 50087.2us
I20251215 08:59:25.457237 23515 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.445189 23325 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.462370 23339 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.463194 23325 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.463935 23284 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:25.464380 23339 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.464445 23449 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:25.448513 23283 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.471678 23283 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:25.464445 23284 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.480301 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:25.476295 21229 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:25.484414 23231 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:25.484895 23231 raft_consensus.cc:697] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:25.485395 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:25.448091 23445 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:25.485817 23231 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.489675 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:25.490975 21230 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.954s (SENT)
I20251215 08:59:25.491417 21230 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:25.493129 23157 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:25.493592 23157 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:25.494057 23157 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:25.501147 23407 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:25.501784 23407 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.503901 23407 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:25.515403 21944 leader_election.cc:336] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.214s (SENT)
I20251215 08:59:25.515972 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:25.516973 23375 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:25.517853 23129 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:25.519199 21264 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
W20251215 08:59:25.522322 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 253474us
I20251215 08:59:25.523757 21379 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:25.523857 23284 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:25.524565 21944 leader_election.cc:336] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.595s (SENT)
I20251215 08:59:25.525173 23150 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251215 08:59:25.525995 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 46619.3us
I20251215 08:59:25.527293 23350 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:25.527360 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:25.528173 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:25.528645 23200 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:25.529352 23200 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:25.529408 23448 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:25.530074 23200 raft_consensus.cc:697] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
W20251215 08:59:25.533309 21407 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1610) took 1534 ms (client timeout 1945 ms). Trace:
W20251215 08:59:25.535576 21407 rpcz_store.cc:269] 1215 08:59:23.998542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:24.241196 (+242654us) service_pool.cc:225] Handling call
1215 08:59:25.533083 (+1291887us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b540026d9d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:25.533233 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2073016064}
I20251215 08:59:25.539598 22925 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:25.540002 22925 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.555310 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:25.560997 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:25.562723 23449 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:25.562808 23157 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.563446 23449 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.556161 23423 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20251215 08:59:25.575706 21971 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:25.558305 23445 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:25.577008 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:25.578503 21413 raft_consensus.cc:606] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:25.578855 21413 raft_consensus.cc:614] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:25.530962 23200 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:25.581566 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54710.3us
I20251215 08:59:25.582108 23449 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:25.582870 21379 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:25.583629 23449 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:25.585067 23146 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:25.585464 23146 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:25.585801 23146 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:25.564661 23263 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:25.587136 23263 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.589128 23263 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:25.591003 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:25.577853 21229 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:25.590766 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 63492us
I20251215 08:59:25.596047 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:25.596624 21341 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:25.565795 23157 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.607641 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:25.608714 21335 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
W20251215 08:59:25.617597 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 81211.9us
I20251215 08:59:25.609445 21339 raft_consensus.cc:1240] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:25.619999 21307 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:25.621147 21232 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:25.608706 23515 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:25.623479 23284 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:25.626191 23284 raft_consensus.cc:697] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:25.626055 23515 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.627153 23284 consensus_queue.cc:237] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:25.627785 21291 heartbeater.cc:646] Failed to heartbeat to 127.19.163.126:33983 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.19.163.126:33983 timed out after 15.000s (SENT)
I20251215 08:59:25.628144 23515 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:25.628840 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:25.603996 22925 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:25.617437 23375 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
W20251215 08:59:25.659657 23523 log.cc:927] Time spent T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f: Append to log took a long time: real 0.085s user 0.007s sys 0.000s
I20251215 08:59:25.659839 23375 raft_consensus.cc:697] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:25.661728 21230 leader_election.cc:336] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.270s (SENT)
I20251215 08:59:25.664181 21413 raft_consensus.cc:606] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:25.664124 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:25.664755 21413 raft_consensus.cc:614] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:25.665293 21415 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:25.670873 21410 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:25.671561 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:25.672259 21414 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
W20251215 08:59:25.673295 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 64062.4us
I20251215 08:59:25.673749 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:25.673781 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 50953us
I20251215 08:59:25.677220 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:25.677882 23157 raft_consensus.cc:2764] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251215 08:59:25.678172 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:25.670501 23375 consensus_queue.cc:237] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.660808 23173 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:25.670724 23283 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:25.681813 21410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:25.681982 21414 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:25.682626 21415 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:25.682940 21341 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:25.686252 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:25.686012 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 78945.7us
W20251215 08:59:25.680585 21944 leader_election.cc:336] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.911s (SENT)
I20251215 08:59:25.692981 21944 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:25.687258 21232 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:25.688421 21410 raft_consensus.cc:606] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:25.699386 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:25.699988 21413 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:25.687794 21411 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:25.698928 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:25.688117 23283 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:25.707636 21335 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:25.698774 23249 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:25.686847 23502 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20251215 08:59:25.714426 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 97010.8us
I20251215 08:59:25.702535 23173 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.718794 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:25.709563 23283 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:25.704934 23157 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:25.708602 23200 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:25.720705 21230 leader_election.cc:336] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.837s (SENT)
I20251215 08:59:25.721375 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:25.721655 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:25.724954 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:25.726703 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 46161us
I20251215 08:59:25.729200 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:25.714987 23249 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:25.733294 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 47151.3us
I20251215 08:59:25.745466 23150 consensus_queue.cc:1059] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:25.746814 23150 consensus_queue.cc:1059] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251215 08:59:25.747476 21379 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:25.747817 23351 raft_consensus.cc:3055] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:25.748179 23351 raft_consensus.cc:740] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:25.748420 21379 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:25.748945 23351 consensus_queue.cc:260] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:25.750064 23351 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:25.752586 21232 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:25.753674 21333 raft_consensus.cc:1240] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:25.753746 21232 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:25.755991 21946 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:25.760859 21341 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251215 08:59:25.769577 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:25.770464 21258 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:25.775238 21410 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:25.736894 23249 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:25.780448 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:25.781672 23268 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:25.782064 23268 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:25.782392 23268 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:25.778435 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:25.786271 23150 raft_consensus.cc:3055] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:25.786669 23150 raft_consensus.cc:740] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:25.787482 23232 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:25.766451 23509 raft_consensus.cc:993] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:25.775177 23181 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:25.775321 23407 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:25.751257 23176 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:25.818390 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:25.787528 23150 consensus_queue.cc:260] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:25.878687 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:25.829531 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 49209.1us
I20251215 08:59:25.819392 23181 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:25.945662 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:25.938634 23540 log.cc:927] Time spent T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
W20251215 08:59:25.882557 23532 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20251215 08:59:25.882910 23508 log.cc:927] Time spent T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251215 08:59:25.934579 23173 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:25.977300 23531 log.cc:927] Time spent T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20251215 08:59:25.977224 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:25.880892 21261 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:25.944346 23445 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:25.880601 23530 log.cc:927] Time spent T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20251215 08:59:25.824530 23488 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:25.947608 23181 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:25.947497 23157 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:25.881279 23403 raft_consensus.cc:1081] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:25.833351 23176 raft_consensus.cc:697] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:25.821909 23407 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:25.881095 23300 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.178s
I20251215 08:59:26.008422 23407 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:26.017169 23157 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.019281 23157 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.019775 23157 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:26.021595 23176 consensus_queue.cc:237] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:26.024364 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:25.996255 23339 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:25.996177 23284 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:26.040077 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 284166us
I20251215 08:59:25.997604 23231 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:25.995277 23150 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:25.828629 23146 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:25.999701 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:26.047955 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 222756us
I20251215 08:59:26.045802 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:26.051812 21261 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:25.996668 23268 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.000231 23445 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.045037 23267 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:26.082432 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.043009 23339 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.047004 23284 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:26.139242 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.224946 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:26.203246 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:26.203635 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:26.115535 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:26.292989 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 213459us
I20251215 08:59:26.301074 21336 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:26.301949 21336 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:26.302479 21336 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:26.302745 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.303748 23284 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.314647 21979 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251215 08:59:26.322089 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 344941us
W20251215 08:59:26.325721 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.316632 21336 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.329044 21336 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.074545 23515 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:26.254309 23283 consensus_meta.cc:287] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.000s
I20251215 08:59:26.187326 23375 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:26.083786 23267 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:26.340059 23540 log.cc:927] Time spent T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.300s user 0.002s sys 0.007s
I20251215 08:59:26.304963 23284 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20251215 08:59:26.207420 23249 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:26.371201 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:26.372299 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.076843 23231 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.389478 23231 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.389539 21261 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:26.369051 23375 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.353856 23515 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.381281 23268 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:26.381956 23232 consensus_meta.cc:287] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.563s user 0.011s sys 0.000s
I20251215 08:59:26.382586 23339 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.383186 23445 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:26.387769 21413 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251215 08:59:26.391883 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.362270 23267 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:26.349030 23283 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.406442 21264 raft_consensus.cc:2393] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:26.414909 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:26.415618 21337 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:26.390348 23146 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:26.417111 21261 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251215 08:59:26.413825 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:26.418285 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:26.420826 21944 leader_election.cc:336] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.630s (SENT)
I20251215 08:59:26.423041 21944 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:26.422210 21261 pending_rounds.cc:85] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 2
I20251215 08:59:26.429561 21261 pending_rounds.cc:107] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251215 08:59:26.429822 21261 raft_consensus.cc:2889] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:26.449618 21333 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:26.450124 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:26.452752 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:26.455107 23445 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:26.458475 21266 raft_consensus.cc:2393] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:26.461280 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:26.424063 23263 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:26.459695 23231 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:26.490813 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:26.468163 23339 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:26.468076 23559 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:26.495543 23268 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:26.527341 23550 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:26.496265 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:26.496775 21410 raft_consensus.cc:1275] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:26.496905 23549 raft_consensus.cc:2764] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20251215 08:59:26.496996 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:26.498210 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:26.510440 23565 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:26.483511 23263 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:26.521826 23557 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.142s user 0.002s sys 0.000s
I20251215 08:59:26.524628 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:26.525660 23564 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:26.530225 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.543537 21339 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:26.535035 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.551579 21381 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:26.565399 21337 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:26.560220 21333 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:26.506382 23303 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:26.566625 21379 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:26.578783 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.586019 21381 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:26.605787 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:26.506745 23231 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:26.625967 21413 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:26.572069 23302 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:26.644490 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:26.648072 21415 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20251215 08:59:26.653789 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 184892us
I20251215 08:59:26.631340 23361 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:26.595824 23232 raft_consensus.cc:2804] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:26.659662 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:26.660176 21335 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 4
I20251215 08:59:26.631860 23249 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.640056 23268 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:26.647679 23557 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20251215 08:59:26.653789 23407 consensus_meta.cc:287] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.637s user 0.010s sys 0.000s
I20251215 08:59:26.662933 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:26.667663 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 246848us
W20251215 08:59:26.672760 21944 leader_election.cc:336] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.717s (SENT)
I20251215 08:59:26.669479 21339 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
W20251215 08:59:26.674176 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.652998 23173 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:26.581298 23303 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.656838 23515 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.651697 23375 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.652709 23544 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:26.587272 23565 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.695230 21340 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:59:26.668818 23283 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.661805 23570 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:26.704466 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.708171 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.711418 21341 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:26.650308 23302 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.586174 23564 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:26.737128 21944 leader_election.cc:336] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.847s (SENT)
I20251215 08:59:26.683287 23407 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.717207 23488 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:26.747225 21264 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:26.747875 21262 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:26.689950 23572 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.067s
I20251215 08:59:26.750320 23303 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:26.754218 23565 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:26.756453 23562 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20251215 08:59:26.757405 23263 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:26.759967 23181 consensus_meta.cc:287] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.750s user 0.011s sys 0.000s
I20251215 08:59:26.733958 23578 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20251215 08:59:26.761582 23231 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:26.761706 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.762989 23302 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:26.763515 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:26.704567 23544 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.767623 23564 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.742012 23570 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:26.769635 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 91717.7us
I20251215 08:59:26.793466 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:26.793790 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 47681.2us
I20251215 08:59:26.777853 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:26.794454 21944 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.023s (SENT)
I20251215 08:59:26.779084 21259 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:26.779601 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.783918 21261 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:26.780002 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.784273 21341 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:26.796211 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.797329 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.797470 23570 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.797878 21341 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:26.798692 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.800486 21338 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:26.801102 21337 raft_consensus.cc:2410] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251215 08:59:26.779675 21976 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251215 08:59:26.803599 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.803889 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.804421 21337 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:59:26.785679 21335 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:26.788264 23544 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:26.747289 21266 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:26.808599 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:26.809772 23303 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:26.810170 23303 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:26.807250 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:26.814123 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.814719 21341 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:26.794399 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:26.816905 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:26.777462 23514 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20251215 08:59:26.817106 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.819141 21335 raft_consensus.cc:2376] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4. Re-sending same reply.
I20251215 08:59:26.820633 21337 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:26.810464 23303 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:26.776973 23522 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20251215 08:59:26.810618 23339 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:26.810503 23407 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.843106 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.841661 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.843715 21340 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:26.843884 21341 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
W20251215 08:59:26.844640 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.783174 23181 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:26.879599 21230 leader_election.cc:336] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.556s (SENT)
W20251215 08:59:26.881109 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:26.880170 21230 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:26.884451 21335 raft_consensus.cc:1240] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:26.886727 21266 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:33840
I20251215 08:59:26.887288 21266 raft_consensus.cc:606] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:26.887651 21266 raft_consensus.cc:614] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:26.888554 21340 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:26.872013 23339 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.891543 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:26.891924 23339 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:26.893625 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:26.894454 23560 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.247s user 0.001s sys 0.000s
I20251215 08:59:26.827492 23555 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.062s
I20251215 08:59:26.822584 23581 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20251215 08:59:26.899050 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1590) took 1145 ms. Trace:
I20251215 08:59:26.901381 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:26.844697 23565 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20251215 08:59:26.898918 23231 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:26.823380 23585 consensus_queue.cc:1048] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.080s
I20251215 08:59:26.899472 21332 rpcz_store.cc:276] 1215 08:59:25.751486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:25.752104 (+ 618us) service_pool.cc:225] Handling call
1215 08:59:25.779731 (+ 27627us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:25.809658 (+ 29927us) raft_consensus.cc:1534] Early marking committed up to index 2
1215 08:59:26.023012 (+213354us) spinlock_profiling.cc:243] Waited 2.38 ms on lock 0x7b1800260da0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b03045 00007f398b5ae317 00007f3987bbe627 00007f3987bcdd3f 00007f3987bcdcb8
1215 08:59:26.121839 (+ 98827us) log.cc:844] Serialized 14 byte log entry
1215 08:59:26.480270 (+358431us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:26.506105 (+ 25835us) log.cc:844] Serialized 52 byte log entry
1215 08:59:26.508465 (+ 2360us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:26.508518 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:26.508622 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:26.893893 (+385271us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:26.896403 (+ 2510us) raft_consensus.cc:1685] finished
1215 08:59:26.896426 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:26.896553 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":115828,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":27729,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":21155,"spinlock_wait_cycles":105602816,"thread_start_us":375366,"threads_started":2,"wal-append.queue_time_us":368560}
I20251215 08:59:26.898803 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:26.920984 21334 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:26.932350 23231 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:26.933933 23176 raft_consensus.cc:3055] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:26.951305 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:26.951903 21333 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:26.947794 23176 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:26.952523 23150 consensus_meta.cc:287] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.906s user 0.008s sys 0.004s
I20251215 08:59:26.937593 23200 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:26.954483 23176 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:26.955265 23303 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.955446 23200 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.957526 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:26.964177 23200 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:26.924556 23181 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:26.965889 21230 leader_election.cc:336] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.519s (SENT)
I20251215 08:59:26.919183 23267 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:26.978832 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:26.983223 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:26.983737 21261 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:26.986167 23576 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
I20251215 08:59:26.986780 23267 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:26.989607 23584 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20251215 08:59:26.986207 23173 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:26.993925 23173 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:26.992672 21261 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:26.994961 23176 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:27.006278 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:27.014675 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.025588 21332 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:27.018442 23200 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
W20251215 08:59:27.034638 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:27.029521 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 376989us
I20251215 08:59:27.037796 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.038424 21336 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:27.039253 23361 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:27.039750 23361 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:27.030933 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 386574us
W20251215 08:59:27.042244 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:27.042800 23361 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:27.043606 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:27.044267 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:27.044833 21979 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:27.031400 23232 raft_consensus.cc:697] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:27.031883 23300 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:27.045188 21378 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:27.046641 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.046690 23514 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:27.047127 23403 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:27.047585 23403 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.053702 21337 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:27.054165 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.054813 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:27.055193 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.055466 21413 raft_consensus.cc:606] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:27.055951 21413 raft_consensus.cc:614] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:27.056387 21408 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:27.058696 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.060010 21378 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:27.059273 21332 raft_consensus.cc:2376] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4. Re-sending same reply.
I20251215 08:59:27.050725 23232 consensus_queue.cc:237] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.052011 23200 raft_consensus.cc:697] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:27.075047 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:27.076067 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.067705 23403 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:27.068665 23267 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:27.069120 23423 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:27.063800 23350 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:27.083732 21409 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:27.084576 21409 raft_consensus.cc:606] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:27.086690 23267 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.060962 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.072357 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.081871 23423 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:27.075523 21979 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:27.071786 21412 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:27.091765 21378 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:27.093883 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.094435 21406 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:27.095005 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.095923 23550 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.096407 23550 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.097483 21381 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:27.098489 23550 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:27.098747 23567 log.cc:927] Time spent T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.467s user 0.005s sys 0.000s
I20251215 08:59:27.098953 21381 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:27.089732 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251215 08:59:27.100597 23361 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:27.101063 23361 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:27.101476 23361 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:27.090369 23423 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.117236 21408 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:27.118682 21406 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:27.121116 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.123143 21412 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:27.125172 21381 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:27.126560 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.089974 21411 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.126165 23350 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:27.129855 23200 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.130239 23173 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.130650 23500 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:27.133550 23173 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:27.135705 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:27.141139 22925 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:27.143579 21413 raft_consensus.cc:2368] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:27.146610 21381 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:27.089000 23267 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:27.155882 21944 leader_election.cc:336] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.599s (SENT)
I20251215 08:59:27.170846 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.125825 23529 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:27.193934 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.194738 21413 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:27.204293 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 1712) took 1017 ms. Trace:
I20251215 08:59:27.204849 21259 rpcz_store.cc:276] 1215 08:59:26.187201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:26.188260 (+ 1059us) service_pool.cc:225] Handling call
1215 08:59:27.204058 (+1015798us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400358254. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:27.204217 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1123981568}
I20251215 08:59:27.197255 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:27.191607 23589 log.cc:927] Time spent T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251215 08:59:27.182206 23529 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:27.216589 21410 raft_consensus.cc:1240] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:27.190879 23550 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:27.222678 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.202262 23587 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:27.223126 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.225208 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.225970 21259 raft_consensus.cc:2368] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6.
I20251215 08:59:27.227738 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:27.229857 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.233608 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.236244 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.236908 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.234547 23181 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:27.238852 23590 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.205s user 0.002s sys 0.000s
I20251215 08:59:27.218847 23529 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:27.234956 23361 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.270216 21381 leader_election.cc:400] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6.
I20251215 08:59:27.272087 21381 leader_election.cc:403] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:27.230057 23550 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.273687 21336 raft_consensus.cc:1275] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:27.274818 23550 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:27.229498 23558 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.276750 21337 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:27.278534 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:27.279062 21979 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:27.280064 21304 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:27.249831 23283 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:27.287659 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:27.287459 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:27.290930 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.289321 23350 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 6
I20251215 08:59:27.292640 23306 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:27.287233 23550 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:27.293838 23550 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:27.294193 23550 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:27.237344 23587 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.298241 23361 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:27.298666 21334 raft_consensus.cc:3055] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:27.298902 21263 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:27.299079 23587 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:27.299557 23303 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:27.271065 23181 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.275947 23231 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.299016 21334 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:27.304842 21410 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251215 08:59:27.245841 23423 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.314093 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.315214 21336 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:27.317499 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.318133 21339 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:27.318766 21408 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:27.319955 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:27.322723 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.322813 23423 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:27.323357 21335 raft_consensus.cc:2376] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4. Re-sending same reply.
I20251215 08:59:27.318703 23249 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.324275 23249 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.326875 23249 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:27.330070 23568 log.cc:927] Time spent T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.609s user 0.001s sys 0.001s
I20251215 08:59:27.331350 23231 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.331700 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.334057 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 198) took 1282 ms. Trace:
I20251215 08:59:27.338244 21407 rpcz_store.cc:276] 1215 08:59:26.051811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:26.061358 (+ 9547us) service_pool.cc:225] Handling call
1215 08:59:26.290779 (+229421us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:26.364509 (+ 73730us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:26.364558 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:26.373492 (+ 8934us) log.cc:844] Serialized 26 byte log entry
1215 08:59:26.566218 (+192726us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:26.586898 (+ 20680us) log.cc:844] Serialized 14 byte log entry
1215 08:59:26.632611 (+ 45713us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:26.632682 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:27.029227 (+396545us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b18002d58c0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984a7b33a 00007f398e1005c9 00007f398ba8982f 00007f3984b06161 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd
1215 08:59:27.329533 (+300306us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:27.333794 (+ 4261us) raft_consensus.cc:1685] finished
1215 08:59:27.333817 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:27.333957 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65933,"prepare.queue_time_us":15192,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":55882,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":12530,"spinlock_wait_cycles":39487488,"thread_start_us":151155,"threads_started":1,"wal-append.queue_time_us":271750}
I20251215 08:59:27.334486 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.300612 23283 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.334683 23181 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:27.320533 21979 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:27.354064 21266 raft_consensus.cc:1275] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:27.355966 23231 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:27.335182 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:27.359431 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:27.361138 21266 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:27.354794 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.345037 21334 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.362195 21338 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
W20251215 08:59:27.368517 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:27.391520 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.393539 21334 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.395095 23578 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.395669 23578 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.290201 23558 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:27.419795 21230 leader_election.cc:336] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.135s (SENT)
W20251215 08:59:27.401051 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:27.416625 21334 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:27.411533 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:27.425350 21263 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.431353 21266 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:27.445334 21381 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:27.427263 21229 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:27.396974 23350 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6.
I20251215 08:59:27.397665 23578 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:27.491689 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.450585 23231 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.464886 23302 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:27.495807 23231 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.494692 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:27.420100 23232 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20251215 08:59:27.506443 23302 raft_consensus.cc:697] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:27.516777 23302 consensus_queue.cc:237] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.528445 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:27.529572 21266 raft_consensus.cc:2393] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:27.533960 21332 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:27.534570 21415 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:27.539256 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:27.540695 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:27.546455 21333 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:27.438475 23550 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:27.547607 21263 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:27.552402 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.488641 23181 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.561988 23283 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:27.563246 23558 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:27.565971 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.567175 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.568073 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.568748 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:27.569160 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 149394us
I20251215 08:59:27.569903 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.570631 21411 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:27.571038 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.571157 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.571606 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.572616 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:27.570675 21230 leader_election.cc:336] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.756s (SENT)
I20251215 08:59:27.575647 21230 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:27.576309 23578 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20251215 08:59:27.576635 21230 leader_election.cc:336] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.246s (SENT)
I20251215 08:59:27.578200 21414 raft_consensus.cc:1217] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[3.5-5.6] Dedup: 5.6->[]
I20251215 08:59:27.578735 21414 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20251215 08:59:27.578256 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.581735 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.581899 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.582540 21409 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:27.583380 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.583880 21409 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:27.585711 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.586359 21412 raft_consensus.cc:2368] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
W20251215 08:59:27.590979 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 116013us
I20251215 08:59:27.553745 23407 raft_consensus.cc:2764] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251215 08:59:27.592892 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:27.592978 21381 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:27.592576 21379 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:27.594426 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.600286 21264 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:27.579905 22925 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:27.604812 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.606395 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.594471 23014 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:27.591921 23556 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:27.613991 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 83472.5us
I20251215 08:59:27.612488 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:27.621898 21230 leader_election.cc:336] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.826s (SENT)
I20251215 08:59:27.592566 23231 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:27.617655 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.629352 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.631217 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.632118 21332 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:27.637449 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.611507 23366 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:27.641638 21406 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:27.591624 23181 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.612316 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:27.650102 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 49479.8us
I20251215 08:59:27.650612 21264 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:27.618803 23014 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:27.655898 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.636101 22927 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:27.661368 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.660732 21332 raft_consensus.cc:1275] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:27.660523 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:27.714995 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 43920.2us
I20251215 08:59:27.690541 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.686508 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.671481 21412 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:27.722596 21334 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:27.692153 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.723304 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.639736 23556 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.726389 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:27.720156 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:27.724906 21409 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:27.727820 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:27.728586 21406 raft_consensus.cc:3055] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:27.728973 21406 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:27.727833 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.730206 21409 raft_consensus.cc:2393] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:27.696708 22927 raft_consensus.cc:697] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:27.732121 22927 consensus_queue.cc:237] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.734781 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.659534 23014 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.729744 21406 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.738346 21266 raft_consensus.cc:1240] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:27.655218 23366 raft_consensus.cc:697] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:27.749186 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.752031 21263 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:59:27.753850 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:27.754426 21259 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:27.752154 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.751107 21260 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:27.671836 23303 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
W20251215 08:59:27.750298 21307 connection.cc:232] RPC call timeout handler was delayed by -3.5407857719999356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
I20251215 08:59:27.764168 23550 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:27.764806 23607 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20251215 08:59:27.765395 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.766151 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.766551 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.726591 23200 consensus_queue.cc:1059] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20251215 08:59:27.752691 23608 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.767604 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.767750 23608 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:27.768250 21340 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:27.770581 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.771986 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.773453 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.774411 21336 raft_consensus.cc:2393] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:27.766702 21337 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.772073 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.776644 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.761525 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.779752 21332 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:27.757313 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 134304us
I20251215 08:59:27.758267 21406 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.785732 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.785944 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.786355 21335 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:27.786652 21336 raft_consensus.cc:2368] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:27.790400 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.752419 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:27.753252 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.793517 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.795728 21412 raft_consensus.cc:2393] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:27.787967 21230 leader_election.cc:336] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.177s (SENT)
I20251215 08:59:27.812041 21332 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:27.825697 21406 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:27.829458 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:27.830448 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 76039.5us
I20251215 08:59:27.831888 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.832940 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.833536 21332 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:27.833451 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.793846 21410 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:27.836122 21335 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:27.836974 21379 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:27.836680 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.847127 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.847759 21332 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:27.848074 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.851501 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.851734 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:27.853667 21410 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:27.855592 23617 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251215 08:59:27.859825 21259 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:27.860385 21337 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:27.862790 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:27.864347 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 76430.9us
I20251215 08:59:27.870543 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.870857 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:27.872112 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.873107 21335 raft_consensus.cc:2368] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:27.872983 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.873761 21336 raft_consensus.cc:2376] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4. Re-sending same reply.
I20251215 08:59:27.829382 23488 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:27.876595 21258 raft_consensus.cc:1240] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20251215 08:59:27.878767 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:27.883708 21332 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:27.886588 21412 raft_consensus.cc:1240] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:27.858661 23173 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:27.894951 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.950759 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.967267 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:27.967869 21335 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:27.974716 21258 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:27.975873 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.964531 21338 raft_consensus.cc:2376] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5. Re-sending same reply.
I20251215 08:59:27.979696 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.985668 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:27.989876 21338 raft_consensus.cc:2393] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:27.966406 23173 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:28.006754 23173 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:28.007181 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 157405us
I20251215 08:59:28.007292 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:28.008203 21341 raft_consensus.cc:2393] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:27.966732 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:28.009208 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:28.009517 21974 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:28.010267 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:28.010780 21341 raft_consensus.cc:2376] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4. Re-sending same reply.
I20251215 08:59:28.011559 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:28.012535 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:28.013175 21336 raft_consensus.cc:2376] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5. Re-sending same reply.
I20251215 08:59:28.012965 23173 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:28.014067 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:28.015085 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:28.015986 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:28.016140 23173 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:28.021638 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:28.022292 21337 raft_consensus.cc:2376] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6. Re-sending same reply.
I20251215 08:59:28.023371 21337 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:28.024703 21337 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20251215 08:59:27.869189 23366 consensus_queue.cc:237] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.026047 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:28.026808 21261 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:28.028364 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:28.028919 21260 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:28.029141 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:28.030282 21259 raft_consensus.cc:1240] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:28.030347 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:28.034600 21335 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:28.041255 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:28.041754 21258 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:28.038785 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:28.052305 21336 raft_consensus.cc:1217] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
I20251215 08:59:28.053046 21336 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251215 08:59:28.043346 21338 raft_consensus.cc:2376] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7. Re-sending same reply.
I20251215 08:59:28.057039 21338 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:28.058959 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:27.987634 23181 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:28.031849 23529 consensus_meta.cc:287] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.725s user 0.010s sys 0.000s
I20251215 08:59:28.066280 21338 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251215 08:59:28.060482 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 189677us
W20251215 08:59:28.059779 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 51441.6us
I20251215 08:59:28.062074 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:28.061456 21332 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251215 08:59:28.064978 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:28.074373 21261 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:28.066255 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:28.075416 21946 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:27.894874 23488 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:28.062613 23608 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:28.112017 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:28.114545 21338 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:28.108091 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 272061us
I20251215 08:59:28.109251 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:28.107923 21332 raft_consensus.cc:1275] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:28.121554 23565 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:28.138559 21944 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:28.062072 23200 consensus_queue.cc:1048] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20251215 08:59:28.041165 23613 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.121s
I20251215 08:59:28.107590 23556 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:28.155947 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 83847.5us
I20251215 08:59:28.110427 23615 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:28.108942 23614 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:28.314492 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 176045us
W20251215 08:59:28.283188 23014 consensus_meta.cc:287] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.530s user 0.007s sys 0.000s
W20251215 08:59:28.283634 23611 log.cc:927] Time spent T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20251215 08:59:28.109896 23303 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:28.070853 23529 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.287469 21974 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:28.284268 23610 consensus_queue.cc:1059] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20251215 08:59:28.274169 23612 log.cc:927] Time spent T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.162s user 0.000s sys 0.000s
W20251215 08:59:28.317386 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:28.416951 21230 leader_election.cc:336] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.660s (SENT)
W20251215 08:59:28.418136 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 297361us
I20251215 08:59:28.419061 21232 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:28.419941 21232 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:28.366904 23608 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:28.424399 21232 leader_election.cc:400] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:28.425424 21413 raft_consensus.cc:2368] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:28.424973 21232 leader_election.cc:403] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20251215 08:59:28.431667 21230 leader_election.cc:336] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.675s (SENT)
W20251215 08:59:28.439881 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 308876us
I20251215 08:59:28.409801 23550 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:28.445554 21230 leader_election.cc:336] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.530s (SENT)
I20251215 08:59:28.424541 23587 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:28.426844 23610 consensus_queue.cc:1243] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer 195a56ac72a84018880bfb3bf6e6ef1d log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20251215 08:59:28.460168 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 45603.3us
I20251215 08:59:28.424377 23556 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:28.396847 23014 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:28.331308 23615 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.482136 21338 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:28.432660 23231 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:28.490322 21339 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.012s sys 0.016s
W20251215 08:59:28.492224 21339 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s user 0.012s sys 0.016s
I20251215 08:59:28.416121 23303 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.429164 23578 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20251215 08:59:28.430935 23608 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.462263 23550 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:28.558458 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 76032.7us
I20251215 08:59:28.590121 23626 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:28.590839 23626 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:28.591317 23626 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:28.578747 23621 consensus_queue.cc:1059] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.187s
I20251215 08:59:28.578608 23618 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:28.598888 23618 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:28.599730 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:28.600386 21973 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:28.578449 23630 raft_consensus.cc:2764] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
W20251215 08:59:28.602916 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 187015us
W20251215 08:59:28.621484 21230 leader_election.cc:336] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.506s (SENT)
W20251215 08:59:28.651983 21260 consensus_meta.cc:287] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.611s user 0.010s sys 0.002s
I20251215 08:59:28.652762 21260 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:28.550978 23488 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:28.688800 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 225) took 1397 ms. Trace:
I20251215 08:59:28.694278 21333 rpcz_store.cc:276] 1215 08:59:27.291713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.291855 (+ 142us) service_pool.cc:225] Handling call
1215 08:59:27.534651 (+242796us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:27.546880 (+ 12229us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.688497 (+1141617us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54001243d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:28.688712 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537598336}
I20251215 08:59:28.722146 23617 raft_consensus.cc:3055] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:28.722723 23617 raft_consensus.cc:740] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:28.723694 23617 consensus_queue.cc:260] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.729358 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1408) took 1166 ms. Trace:
I20251215 08:59:28.729735 21414 rpcz_store.cc:276] 1215 08:59:27.562941 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.571980 (+ 9039us) service_pool.cc:225] Handling call
1215 08:59:27.578087 (+ 6107us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:27.581475 (+ 3388us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.729132 (+1147657us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b54000bca50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:28.729280 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13148416}
I20251215 08:59:28.734607 21263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1379) took 1142 ms. Trace:
I20251215 08:59:28.735047 21263 rpcz_store.cc:276] 1215 08:59:27.592375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.593389 (+ 1014us) service_pool.cc:225] Handling call
1215 08:59:28.734369 (+1140980us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b540020ab54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:28.734523 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":336656768}
I20251215 08:59:28.737531 21407 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:28.738242 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1410) took 1174 ms. Trace:
I20251215 08:59:28.738554 21407 rpcz_store.cc:276] 1215 08:59:27.563533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.572032 (+ 8499us) service_pool.cc:225] Handling call
1215 08:59:27.773856 (+201824us) spinlock_profiling.cc:243] Waited 9.33 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:28.737051 (+963195us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8dec 00007f3984ac89ab 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:28.737300 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:28.737981 (+ 681us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.738184 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67718144}
I20251215 08:59:28.742260 23303 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:28.742991 23303 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:28.743614 23303 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:28.748790 23623 log.cc:927] Time spent T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20251215 08:59:28.705835 23638 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:28.709664 23621 raft_consensus.cc:3055] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:28.801030 23621 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:28.801963 23621 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.749965 23614 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:28.803449 23614 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.805562 23614 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:28.806411 23014 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:28.750628 23529 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:28.797796 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1628) took 1058 ms. Trace:
I20251215 08:59:28.807560 21334 rpcz_store.cc:276] 1215 08:59:27.738975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.739307 (+ 332us) service_pool.cc:225] Handling call
1215 08:59:27.756122 (+ 16815us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:27.782723 (+ 26601us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:27.782766 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:27.782826 (+ 60us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:27.782880 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.797491 (+1014611us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b5400164c54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:28.797555 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:28.797724 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60714240}
I20251215 08:59:28.811383 21411 raft_consensus.cc:1240] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:28.812106 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1413) took 1247 ms. Trace:
I20251215 08:59:28.811780 23638 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:28.814745 21265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1751) took 1141 ms. Trace:
I20251215 08:59:28.815420 23550 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:28.816449 23615 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:28.815109 21265 rpcz_store.cc:276] 1215 08:59:27.673668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.675420 (+ 1752us) service_pool.cc:225] Handling call
1215 08:59:28.739757 (+1064337us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:28.739946 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:28.739974 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:28.740030 (+ 56us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:59:28.740070 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.740128 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:28.814538 (+ 74410us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540020ab50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:28.814687 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2398373248}
I20251215 08:59:28.812433 21411 rpcz_store.cc:276] 1215 08:59:27.564606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.572488 (+ 7882us) service_pool.cc:225] Handling call
1215 08:59:28.810909 (+1238421us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:28.811136 (+ 227us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:28.811849 (+ 713us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.812036 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10435072}
I20251215 08:59:29.000861 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1633) took 1237 ms. Trace:
W20251215 08:59:28.834673 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 392666us
I20251215 08:59:29.153443 21335 rpcz_store.cc:276] 1215 08:59:27.763205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:28.056682 (+293477us) service_pool.cc:225] Handling call
1215 08:59:28.059019 (+ 2337us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:28.145631 (+ 86612us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:28.145674 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:28.145727 (+ 53us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:28.145786 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:29.000513 (+854727us) spinlock_profiling.cc:243] Waited 86.4 ms on lock 0x7b5400278dd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:29.000575 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:29.000761 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198642048}
I20251215 08:59:29.154904 21232 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:29.155311 23621 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:29.155613 23638 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:29.134660 23556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.007s sys 0.011s
W20251215 08:59:29.158565 23556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.008s sys 0.013s
I20251215 08:59:29.159003 23556 raft_consensus.cc:697] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:29.159111 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1750) took 1503 ms. Trace:
I20251215 08:59:29.044972 23641 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:29.160238 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1416) took 1577 ms. Trace:
I20251215 08:59:28.870152 21972 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 571) took 1008 ms. Trace:
I20251215 08:59:29.171329 21972 rpcz_store.cc:276] 1215 08:59:27.861639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.862720 (+ 1081us) service_pool.cc:225] Handling call
1215 08:59:28.868367 (+1005647us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:28.868626 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:28.868920 (+ 294us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:28.868966 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:28.869734 (+ 768us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:28.869798 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.869886 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:28.870068 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":179833728}
I20251215 08:59:29.158396 23618 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:28.847913 21258 consensus_meta.cc:287] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.797s user 0.009s sys 0.002s
I20251215 08:59:29.187086 21258 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:29.187732 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1393) took 1203 ms. Trace:
I20251215 08:59:29.160970 23641 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.045229 23267 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:29.160682 21408 rpcz_store.cc:276] 1215 08:59:27.582860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.594990 (+ 12130us) service_pool.cc:225] Handling call
1215 08:59:27.606485 (+ 11495us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:27.636437 (+ 29952us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:27.636465 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:27.645310 (+ 8845us) log.cc:844] Serialized 26 byte log entry
1215 08:59:27.845320 (+200010us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:27.884862 (+ 39542us) log.cc:844] Serialized 14 byte log entry
1215 08:59:28.014914 (+130052us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b180034b820. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b03045 00007f3987babce0 00007f3987bbe396 00007f3987bcef07 00007f3987bcee88
1215 08:59:28.114814 (+ 99900us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:28.114938 (+ 124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:28.516442 (+401504us) raft_consensus.cc:1685] finished
1215 08:59:28.516539 (+ 97us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:29.160178 (+643639us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45685,"prepare.queue_time_us":39354,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":1885,"raft.queue_time_us":150009,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":46484,"spinlock_wait_cycles":11428608,"thread_start_us":253303,"threads_started":2,"wal-append.queue_time_us":390503}
I20251215 08:59:28.908490 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59102 (request call id 204) took 1696 ms. Trace:
I20251215 08:59:29.189374 21413 rpcz_store.cc:276] 1215 08:59:27.211954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.212164 (+ 210us) service_pool.cc:225] Handling call
1215 08:59:28.908210 (+1696046us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54002945d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:28.908402 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2763222400}
I20251215 08:59:29.164755 23550 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:28.829726 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.165279 23366 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:29.166986 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 545516us
I20251215 08:59:29.192415 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.170542 23488 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.192518 23366 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.174306 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:29.192106 21409 raft_consensus.cc:2410] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20251215 08:59:29.174080 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:29.175932 21338 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:29.194706 23366 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.193315 23488 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.159654 21264 rpcz_store.cc:276] 1215 08:59:27.655786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.655977 (+ 191us) service_pool.cc:225] Handling call
1215 08:59:27.656403 (+ 426us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:27.663314 (+ 6911us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:27.663362 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:27.664084 (+ 722us) log.cc:844] Serialized 26 byte log entry
1215 08:59:27.758979 (+ 94895us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:27.759041 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:27.759159 (+ 118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:28.497804 (+738645us) raft_consensus.cc:1685] finished
1215 08:59:28.497832 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:29.159033 (+661201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6245,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"thread_start_us":94406,"threads_started":1,"wal-append.queue_time_us":337319}
I20251215 08:59:29.188100 21258 rpcz_store.cc:276] 1215 08:59:27.984643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.989224 (+ 4581us) service_pool.cc:225] Handling call
1215 08:59:29.187678 (+1198454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:29.193183 21337 raft_consensus.cc:2368] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:29.193763 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1426) took 1588 ms. Trace:
I20251215 08:59:29.198072 21409 rpcz_store.cc:276] 1215 08:59:27.605164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.745486 (+140322us) service_pool.cc:225] Handling call
1215 08:59:28.829513 (+1084027us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:29.193717 (+364204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12795648}
I20251215 08:59:29.159190 23617 raft_consensus.cc:3060] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:29.196113 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:29.201809 21338 raft_consensus.cc:2376] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7. Re-sending same reply.
I20251215 08:59:29.203478 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.204342 21338 raft_consensus.cc:2393] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:29.160028 23556 consensus_queue.cc:237] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.192551 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:29.199976 21164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36406 (request call id 227) took 18611 ms (client timeout 15000 ms). Trace:
I20251215 08:59:29.169185 23608 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.196019 23618 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.200074 23302 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.232491 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.238502 21335 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:29.211335 23403 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.206362 23638 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:29.227464 23608 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.211506 23303 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.197855 23550 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.241371 21162 catalog_manager.cc:5654] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:29.208107 23267 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.272735 21162 catalog_manager.cc:5654] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:29.274401 21162 catalog_manager.cc:5654] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:29.276561 21162 catalog_manager.cc:5654] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 3, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 3 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:29.240558 23618 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.241806 23302 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:29.286303 21378 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.012s sys 0.024s
W20251215 08:59:29.286706 21378 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.776s user 0.012s sys 0.024s
W20251215 08:59:29.280686 21379 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.013s sys 0.023s
W20251215 08:59:29.288518 21379 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.874s user 0.013s sys 0.023s
W20251215 08:59:29.289391 21946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.815s user 0.004s sys 0.036s
W20251215 08:59:29.289765 21946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.004s sys 0.036s
W20251215 08:59:29.291944 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 843974us
I20251215 08:59:29.247584 23403 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.296078 21304 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:29.301662 23635 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:29.302213 23403 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.302471 23635 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.303162 23641 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.304126 23488 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.304638 23488 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.305133 23488 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.305823 23635 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:29.306908 23544 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.307550 23544 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.267231 23608 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.285974 23632 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.309424 23632 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.310082 23544 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.311168 23635 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:29.311863 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.312837 21338 raft_consensus.cc:2393] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:29.313764 21162 catalog_manager.cc:5654] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:29.291663 21341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.846s user 0.025s sys 0.035s
W20251215 08:59:29.316286 21341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s user 0.025s sys 0.035s
I20251215 08:59:29.264776 23638 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:29.315730 21306 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.844s user 0.014s sys 0.050s
W20251215 08:59:29.319978 21944 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.010s sys 0.014s
W20251215 08:59:29.320993 21944 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.010s sys 0.014s
I20251215 08:59:29.312389 23608 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.307240 23488 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.311199 23632 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:29.205804 21164 rpcz_store.cc:269] 1215 08:59:10.589045 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:10.591515 (+ 2470us) service_pool.cc:225] Handling call
1215 08:59:29.199818 (+18608303us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:25.288591 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:25.310365 (+ 21774us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:25.310422 (+ 57us) write_op.cc:435] Acquired schema lock
1215 08:59:25.310461 (+ 39us) tablet.cc:660] Decoding operations
1215 08:59:25.318183 (+ 7722us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:25.318284 (+ 101us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:25.318331 (+ 47us) tablet.cc:683] Acquiring locks for 19 operations
1215 08:59:25.409854 (+ 91523us) tablet.cc:699] Row locks acquired
1215 08:59:25.409888 (+ 34us) write_op.cc:260] PREPARE: finished
1215 08:59:25.410072 (+ 184us) write_op.cc:270] Start()
1215 08:59:25.410261 (+ 189us) write_op.cc:276] Timestamp: P: 1765789165409988 usec, L: 0
1215 08:59:25.410298 (+ 37us) op_driver.cc:348] REPLICATION: starting
1215 08:59:25.410993 (+ 695us) log.cc:844] Serialized 7609 byte log entry
1215 08:59:25.721935 (+310942us) op_driver.cc:464] REPLICATION: finished
1215 08:59:25.836638 (+114703us) write_op.cc:301] APPLY: starting
1215 08:59:25.836734 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:25.877670 (+ 40936us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:25.879694 (+ 2024us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:25.972673 (+ 92979us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:25.983553 (+ 10880us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=19
1215 08:59:25.983623 (+ 70us) write_op.cc:312] APPLY: finished
1215 08:59:25.984089 (+ 466us) log.cc:844] Serialized 132 byte log entry
1215 08:59:25.984297 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:25.984500 (+ 203us) write_op.cc:454] Released schema lock
1215 08:59:25.984571 (+ 71us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:25.984980 (+ 409us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:25.985035 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10096,"reported_tablets":34,"child_traces":[["op",{"apply.queue_time_us":100157,"apply.run_cpu_time_us":12304,"apply.run_wall_time_us":148358,"num_ops":19,"prepare.queue_time_us":226686,"prepare.run_cpu_time_us":35818,"prepare.run_wall_time_us":281255,"raft.queue_time_us":60840,"raft.run_cpu_time_us":4399,"raft.run_wall_time_us":22227,"replication_time_us":311486,"thread_start_us":287356,"threads_started":4,"wal-append.queue_time_us":162624,"wal-append.run_cpu_time_us":17923,"wal-append.run_wall_time_us":1411825}]]}
W20251215 08:59:29.319830 21306 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.848s user 0.014s sys 0.050s
W20251215 08:59:29.343601 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:29.346777 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.347500 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.348790 23488 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:29.364732 23488 raft_consensus.cc:697] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:29.360110 23302 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:29.361800 23649 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.367174 23267 leader_election.cc:290] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:29.369827 23649 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:29.362489 21973 consensus_meta.cc:287] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.762s user 0.007s sys 0.003s
W20251215 08:59:29.320302 23407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.017s sys 0.007s
I20251215 08:59:29.362881 23652 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.366029 23618 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.367162 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:29.311863 23635 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:29.332551 23231 consensus_meta.cc:287] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.842s user 0.009s sys 0.002s
W20251215 08:59:29.354766 21229 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.031s sys 0.035s
I20251215 08:59:29.376494 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:29.377709 21335 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251215 08:59:29.379696 21979 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:29.374161 21162 catalog_manager.cc:5654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 3, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 3 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:29.399633 21229 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.031s sys 0.035s
I20251215 08:59:29.400796 21337 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:29.376421 23407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.870s user 0.017s sys 0.007s
I20251215 08:59:29.426689 21162 catalog_manager.cc:5654] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:29.384254 23630 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.006s sys 0.012s
I20251215 08:59:29.474606 21335 raft_consensus.cc:3055] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:29.475073 21335 raft_consensus.cc:740] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:29.384394 23587 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.885s user 0.005s sys 0.013s
I20251215 08:59:29.474335 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:29.376186 21973 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:29.476699 21162 catalog_manager.cc:5654] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 2, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 2 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:29.497709 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.550603 21334 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:29.475968 21335 consensus_queue.cc:260] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:29.476747 23630 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.695s user 0.006s sys 0.013s
I20251215 08:59:29.384155 23652 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.526209 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.437516 23231 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:29.525523 21340 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:29.529269 23550 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:29.487721 23587 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.992s user 0.007s sys 0.019s
I20251215 08:59:29.571480 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.537153 21979 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:29.543803 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:29.572373 21334 raft_consensus.cc:2393] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:29.539676 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.525511 23654 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
W20251215 08:59:29.493069 21945 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.011s sys 0.013s
I20251215 08:59:29.551059 21162 catalog_manager.cc:5654] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 2 to 5, leader changed from f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:29.552683 21335 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:29.555670 21332 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:29.538667 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1636) took 1774 ms. Trace:
I20251215 08:59:29.580453 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:29.581934 21230 leader_election.cc:336] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.015s (SENT)
I20251215 08:59:29.562341 23652 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.564492 23545 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:29.565285 23656 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:29.594751 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.565519 21978 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:29.595660 21334 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:29.565826 23231 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.567245 23658 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:29.568521 21340 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:29.597935 21337 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:29.598537 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.599105 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.569304 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:29.571089 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:29.579960 21973 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 509) took 2194 ms (client timeout 2092 ms). Trace:
I20251215 08:59:29.573319 21333 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:29.572521 21977 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:29.545965 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:29.575906 21944 leader_election.cc:336] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.012s (SENT)
W20251215 08:59:29.582967 21230 leader_election.cc:336] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.103s (SENT)
I20251215 08:59:29.580749 21336 rpcz_store.cc:276] 1215 08:59:27.764102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:28.053882 (+289780us) service_pool.cc:225] Handling call
1215 08:59:28.054196 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:28.054439 (+ 243us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:28.070709 (+ 16270us) log.cc:844] Serialized 14 byte log entry
1215 08:59:29.537582 (+1466873us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:29.537635 (+ 53us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:29.537676 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:29.537750 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:29.538601 (+ 851us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15682,"prepare.queue_time_us":16434,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":654929,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"spinlock_wait_cycles":10742656,"thread_start_us":818806,"threads_started":2,"wal-append.queue_time_us":1279009}
I20251215 08:59:29.604139 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.593541 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:29.606745 21230 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:29.608739 21230 leader_election.cc:336] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.196s (SENT)
I20251215 08:59:29.594511 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.613917 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.614473 21332 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:29.600411 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:29.603111 21944 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.247s (SENT)
I20251215 08:59:29.601478 21337 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
W20251215 08:59:29.578438 21945 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.011s sys 0.014s
I20251215 08:59:29.573683 23587 raft_consensus.cc:697] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:29.600400 21973 rpcz_store.cc:269] 1215 08:59:27.361378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.364735 (+ 3357us) service_pool.cc:225] Handling call
1215 08:59:28.599562 (+1234827us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:29.556222 (+956660us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9894144}
I20251215 08:59:29.615499 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:29.578475 23618 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.578295 23302 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:29.555269 23651 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.609686 23653 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.619477 23651 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.619728 23653 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.621678 23651 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:29.611776 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.622884 23662 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.623483 23662 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.587232 22927 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:29.617776 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.621723 23653 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:29.596292 23649 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.595403 23624 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:29.595595 23014 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.625880 21340 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:29.594966 23652 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.617679 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.630092 21336 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:29.630965 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.633401 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.631186 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.597486 23645 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:29.635138 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:29.641549 21337 raft_consensus.cc:2368] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:29.638649 21335 raft_consensus.cc:1275] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251215 08:59:29.642251 21335 pending_rounds.cc:85] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:29.642551 21335 pending_rounds.cc:107] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251215 08:59:29.642812 21335 raft_consensus.cc:2889] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:29.613973 23638 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.557588 23488 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.647593 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.639210 23014 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.648178 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.645408 23181 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:29.648710 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:29.645460 23653 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.652665 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.645356 23652 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.646572 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:29.634963 23544 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.675422 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:29.614264 23658 raft_consensus.cc:697] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:29.618279 23656 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.598013 23545 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:29.635123 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:29.579979 23654 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.660657 21409 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:29.662732 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:29.684669 21264 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
W20251215 08:59:29.664953 21339 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.172s user 0.010s sys 0.043s
I20251215 08:59:29.610386 23231 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.626713 23302 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.626538 23618 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:29.685580 21339 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.193s user 0.010s sys 0.043s
W20251215 08:59:29.656498 23631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.029s sys 0.031s
I20251215 08:59:29.689529 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:29.690758 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1621) took 1928 ms. Trace:
I20251215 08:59:29.656419 21379 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:29.693074 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251215 08:59:29.699613 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 52488.4us
I20251215 08:59:29.630576 22927 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.691223 21339 rpcz_store.cc:276] 1215 08:59:27.762186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.770831 (+ 8645us) service_pool.cc:225] Handling call
1215 08:59:27.771131 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:27.771329 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:27.862165 (+ 90836us) log.cc:844] Serialized 14 byte log entry
1215 08:59:29.690327 (+1828162us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:29.690396 (+ 69us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:29.690462 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:29.690559 (+ 97us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:29.690694 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":86490,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":675179,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":1810566,"threads_started":3,"wal-append.queue_time_us":1460050}
I20251215 08:59:29.646255 23624 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.710767 21946 leader_election.cc:400] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:29.711153 21946 leader_election.cc:403] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:29.712482 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:29.654564 23645 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.686808 23635 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:29.701083 23631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.951s user 0.029s sys 0.031s
I20251215 08:59:29.717033 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:29.724705 21264 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:29.730937 21258 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:29.690626 23608 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:29.736004 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:29.682729 23544 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.714747 23651 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:29.674227 23653 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.703902 23664 raft_consensus.cc:2764] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20251215 08:59:29.712415 23625 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:29.686923 23656 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:29.740235 23643 log.cc:927] Time spent T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.254s user 0.001s sys 0.001s
W20251215 08:59:29.749804 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 46515.3us
I20251215 08:59:29.754925 21307 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:29.759732 21407 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:29.732343 23407 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.787887 23676 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.788609 23676 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.789382 23631 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.789916 23631 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.790560 23649 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.791167 23649 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.791875 23631 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.792416 23631 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.722637 23587 consensus_queue.cc:237] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:29.794361 23666 log.cc:927] Time spent T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20251215 08:59:29.797436 23014 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.799360 23624 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.801571 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:29.803282 22927 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.806152 23302 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:29.807595 23231 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.813237 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:29.741396 23608 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.749653 23670 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:29.755865 23651 raft_consensus.cc:697] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:29.808305 23231 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.763645 23662 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.884622 23676 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.884670 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:29.836220 23621 consensus_meta.cc:287] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.680s user 0.010s sys 0.001s
W20251215 08:59:29.883358 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 65349.8us
I20251215 08:59:29.808395 23652 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.810001 23656 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.892776 23645 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.871529 23664 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:29.897475 23626 consensus_meta.cc:287] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 1.306s user 0.004s sys 0.011s
I20251215 08:59:29.834367 23649 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.787312 23407 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.922627 23626 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.902135 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1634) took 2138 ms. Trace:
I20251215 08:59:29.913275 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:29.913753 23653 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.763005 23658 consensus_queue.cc:237] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.914448 23654 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.888989 23608 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.920718 23618 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:29.942668 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:29.904503 23670 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:29.896840 23231 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.921690 23651 consensus_queue.cc:237] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.903133 23544 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.792882 23631 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:29.937770 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:29.919498 23617 consensus_meta.cc:287] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.709s user 0.007s sys 0.001s
W20251215 08:59:29.941062 21230 leader_election.cc:336] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.884s (SENT)
I20251215 08:59:29.924682 23407 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.904191 23550 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:29.964428 23550 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:29.929253 21341 rpcz_store.cc:276] 1215 08:59:27.763515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:28.055940 (+292425us) service_pool.cc:225] Handling call
1215 08:59:28.056231 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:28.056494 (+ 263us) raft_consensus.cc:1534] Early marking committed up to index 4
1215 08:59:28.427367 (+370873us) log.cc:844] Serialized 14 byte log entry
1215 08:59:29.683668 (+1256301us) log.cc:844] Serialized 14 byte log entry
1215 08:59:29.900526 (+216858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:29.900597 (+ 71us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:29.900669 (+ 72us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:29.901924 (+ 1255us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:29.902069 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55974,"prepare.queue_time_us":51643,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":182574,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":7437,"thread_start_us":1780745,"threads_started":4,"wal-append.queue_time_us":1135268}
I20251215 08:59:29.924796 23626 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:29.965512 23644 log.cc:927] Time spent T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.423s user 0.001s sys 0.004s
I20251215 08:59:29.950819 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:29.932139 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 346325us
I20251215 08:59:29.966149 21973 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:29.967043 23545 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:29.970363 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:29.972486 21229 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:29.973850 23407 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:29.974229 23407 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:29.974615 23407 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:29.980109 21265 raft_consensus.cc:1275] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:29.981217 23631 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:29.982415 23407 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.983978 23231 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.984951 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:29.958034 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:29.962498 23654 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:29.958122 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.003119 21972 raft_consensus.cc:1275] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20251215 08:59:29.995214 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 278711us
I20251215 08:59:29.951301 23670 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:29.991439 23407 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:29.929824 23302 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20251215 08:59:29.993757 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:29.942826 23621 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.046s
W20251215 08:59:29.996413 21230 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.873s (SENT) [suppressed 38 similar messages]
I20251215 08:59:30.001111 21263 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:30.014292 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:30.004571 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:29.958343 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.002805 23654 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:30.009723 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.018558 23670 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:30.031080 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:30.023976 21974 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:30.033092 21229 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:30.009723 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:30.035854 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 269861us
W20251215 08:59:30.048204 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 293340us
I20251215 08:59:30.030848 23654 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:30.049969 21307 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.054344 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.055022 23618 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:30.045794 23181 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:30.057478 23181 raft_consensus.cc:697] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:30.056113 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:30.058424 23181 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.063762 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.064095 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.064596 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.064916 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.065446 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.066099 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.067824 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:30.066282 23618 raft_consensus.cc:2764] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20251215 08:59:30.067447 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.066831 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.069164 23654 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.071017 23618 raft_consensus.cc:2764] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251215 08:59:30.071107 23654 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:30.072086 21307 leader_election.cc:400] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:30.073688 21307 leader_election.cc:403] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251215 08:59:30.075349 23618 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:30.050632 21971 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:30.078403 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.082394 23654 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:30.083484 23654 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.085496 23654 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:30.086544 23550 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:30.088192 23670 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:30.089524 21307 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:30.095937 23550 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:30.096477 23550 raft_consensus.cc:697] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:30.096949 21229 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:30.097393 23550 consensus_queue.cc:237] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.097913 23407 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:30.098403 23407 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:30.098752 23407 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:30.101368 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:30.102185 23684 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:30.102572 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:30.102695 23684 raft_consensus.cc:697] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:30.103739 23684 consensus_queue.cc:237] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.065912 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.113843 23655 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 08:59:30.114251 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:30.114430 23655 raft_consensus.cc:697] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:30.115453 23655 consensus_queue.cc:237] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.120612 23618 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:30.121181 23618 raft_consensus.cc:697] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:30.121618 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 155307us
I20251215 08:59:30.121987 23618 consensus_queue.cc:237] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:30.123253 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 56266.7us
I20251215 08:59:30.123383 21378 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:30.124475 21378 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:30.131426 23014 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:30.131947 23014 raft_consensus.cc:697] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:30.132021 23681 consensus_queue.cc:1059] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:30.132915 23682 raft_consensus.cc:3055] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:30.133471 23682 raft_consensus.cc:740] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:30.132867 23014 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.134240 23682 consensus_queue.cc:260] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.135401 23682 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:30.136801 23645 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251215 08:59:30.139403 23645 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:30.139725 23645 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:30.140097 23645 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:30.141820 23302 consensus_queue.cc:1048] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:30.131419 23670 consensus_queue.cc:1059] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20251215 08:59:30.143097 23670 raft_consensus.cc:3055] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:30.143530 23670 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:30.145203 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 108698us
I20251215 08:59:30.145934 21379 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.146186 21976 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:30.147106 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:30.147869 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.148007 23556 consensus_queue.cc:1059] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251215 08:59:30.149170 23556 raft_consensus.cc:3055] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:30.149566 23556 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:30.149834 22927 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:30.150279 22927 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:30.150728 22927 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:30.153231 23635 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:30.150487 23556 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.153638 23635 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:30.155260 23635 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:30.155722 23556 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:30.144842 23350 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:30.156929 23350 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.159381 23350 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:30.154052 22925 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:30.161311 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:30.163336 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.164166 21972 raft_consensus.cc:2410] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20251215 08:59:30.144464 23670 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.161212 22925 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:30.167588 22925 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:30.167858 23670 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:30.168819 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.169389 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:30.169974 21975 raft_consensus.cc:2376] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5. Re-sending same reply.
I20251215 08:59:30.170112 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.173528 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.174281 21978 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:30.177883 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.179121 23544 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
W20251215 08:59:30.162823 21230 leader_election.cc:336] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.637s (SENT)
I20251215 08:59:30.181191 21230 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:30.179514 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.179631 23544 raft_consensus.cc:697] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:30.189102 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:30.209681 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:30.210758 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 78033.3us
W20251215 08:59:30.211578 21230 leader_election.cc:336] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.656s (SENT)
I20251215 08:59:30.216279 21230 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:30.215459 21307 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.185145 23231 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:30.224249 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:30.241176 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:30.229835 21230 leader_election.cc:336] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.850s (SENT)
I20251215 08:59:30.243855 21230 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:30.244199 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:30.244855 21413 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:30.244920 21230 leader_election.cc:336] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.010s (SENT)
I20251215 08:59:30.245378 21230 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:30.234333 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 77869.5us
I20251215 08:59:30.256687 21381 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.220299 23550 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:30.268919 21407 raft_consensus.cc:1240] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:30.270694 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.226632 23627 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:30.273783 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.274441 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.226457 23403 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:30.277829 22925 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.280112 22925 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:30.281319 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.281646 23700 consensus_queue.cc:1048] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:30.282565 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.283037 23686 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:30.263434 23631 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:30.249768 23658 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:30.306437 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.254068 23621 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:30.307471 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.219475 23407 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.309167 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.309484 23544 consensus_queue.cc:237] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.313452 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:30.318686 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:30.324726 21413 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
W20251215 08:59:30.362803 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:30.388953 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 119120us
W20251215 08:59:30.392529 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 176232us
I20251215 08:59:30.264351 23675 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:30.393313 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:30.392335 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 135716us
I20251215 08:59:30.395339 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.263168 23626 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.062s
I20251215 08:59:30.396687 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.403990 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.386190 23014 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:30.406668 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.274878 23617 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.122s
I20251215 08:59:30.315694 23621 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:30.305629 23627 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.429338 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.424705 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.426462 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.426749 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.385489 23403 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.433243 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.400197 23626 consensus_queue.cc:1243] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer 0a2fcb89589c401d8efc8d86da4ba0f4 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251215 08:59:30.431316 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.398717 23565 consensus_queue.cc:1059] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
W20251215 08:59:30.445744 23695 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251215 08:59:30.413674 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:30.455825 23403 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:30.441607 23653 consensus_queue.cc:1048] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20251215 08:59:30.462976 23627 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:30.465878 23645 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.432380 23621 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:30.471871 21978 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:30.473026 23635 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.474210 23550 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:30.475268 23635 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:30.449232 23587 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:30.450186 23665 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:30.450893 23181 consensus_queue.cc:1048] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:30.423064 23014 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.452248 23231 raft_consensus.cc:3055] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:30.477942 23231 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:30.479081 23014 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:30.478856 23231 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.480120 23231 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:30.453251 23565 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:30.453450 21230 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.481904 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:30.483326 21230 leader_election.cc:400] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5.
I20251215 08:59:30.484256 21230 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:30.485222 23665 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:30.485879 23665 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:30.487236 23688 log.cc:927] Time spent T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20251215 08:59:30.489818 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.490371 21415 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:30.396049 22927 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.485574 23407 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:30.501443 23645 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:30.507184 23692 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20251215 08:59:30.516264 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:30.518328 21307 connection.cc:232] RPC call timeout handler was delayed by -1.5463137819999702s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
W20251215 08:59:30.519827 21307 connection.cc:232] RPC call timeout handler was delayed by -1.5355137819999527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20251215 08:59:30.485271 23621 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:30.522356 21307 connection.cc:232] RPC call timeout handler was delayed by -1.4860137819999863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1572s for a response
I20251215 08:59:30.524705 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:30.525136 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 131118us
W20251215 08:59:30.526652 21307 connection.cc:232] RPC call timeout handler was delayed by -1.4716137819999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20251215 08:59:30.527407 21307 connection.cc:232] RPC call timeout handler was delayed by -1.4653137819999529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
W20251215 08:59:30.533914 21307 connection.cc:232] RPC call timeout handler was delayed by -1.4149137819999851s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
I20251215 08:59:30.535203 21341 raft_consensus.cc:1275] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:30.533424 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:30.536829 21307 connection.cc:232] RPC call timeout handler was delayed by -1.4131137819999822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1653s for a response
W20251215 08:59:30.538272 21307 connection.cc:232] RPC call timeout handler was delayed by -1.3546137820000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
W20251215 08:59:30.538898 21307 connection.cc:232] RPC call timeout handler was delayed by -1.263713781999968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
W20251215 08:59:30.540457 21307 connection.cc:232] RPC call timeout handler was delayed by -1.2412137820000453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1844s for a response
W20251215 08:59:30.541131 21307 connection.cc:232] RPC call timeout handler was delayed by -1.2025137819999827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
W20251215 08:59:30.542058 21307 connection.cc:232] RPC call timeout handler was delayed by -1.0963137820000384s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20251215 08:59:30.542914 21307 connection.cc:232] RPC call timeout handler was delayed by -1.044113781999954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
I20251215 08:59:30.544565 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.545755 21338 raft_consensus.cc:2410] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:30.547726 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.548934 21339 raft_consensus.cc:2410] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251215 08:59:30.552775 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.553869 21341 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:30.555013 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.556618 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.558038 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.565340 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.566040 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.560647 21338 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:30.567266 21307 connection.cc:232] RPC call timeout handler was delayed by -1.0234137820000342s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
I20251215 08:59:30.573645 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.572803 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.572988 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:30.571445 21333 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:30.575153 21338 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:30.572639 21307 connection.cc:232] RPC call timeout handler was delayed by -1.0234137820000342s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
W20251215 08:59:30.576321 21307 connection.cc:232] RPC call timeout handler was delayed by -1.007213782000008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
I20251215 08:59:30.578049 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:30.578605 21336 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:30.579582 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:30.580107 21415 raft_consensus.cc:606] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:30.582857 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:30.594261 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.600602 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.601894 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:30.602826 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.603538 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.604449 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.604950 21406 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:30.606418 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.607897 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.608512 21340 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 1.
I20251215 08:59:30.608745 21307 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:30.609337 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.609927 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.610363 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.613111 21414 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:30.573727 23705 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:30.619235 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.619868 21408 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:30.620815 21411 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:30.629621 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.628788 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.624727 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:30.632855 21339 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:30.629750 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.624974 21336 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:30.645819 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.648090 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.651782 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.621526 23681 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:30.622440 23665 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:30.662585 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:30.654044 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 45382.1us
I20251215 08:59:30.655829 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.656142 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.657573 21415 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:30.657683 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:30.660176 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:30.665894 21332 raft_consensus.cc:2368] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
W20251215 08:59:30.663555 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:30.673187 21338 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:30.674266 21337 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20251215 08:59:30.684985 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 290867us
I20251215 08:59:30.685925 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:30.688751 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:30.660666 23550 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:30.658641 22927 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:30.692690 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:30.695541 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:30.696434 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 167554us
I20251215 08:59:30.695976 21415 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:30.697918 21381 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.634294 23705 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.693257 21412 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "beefe15185ac42ef9a20251007f39a56"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:30.695758 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.700467 21412 raft_consensus.cc:606] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS beefe15185ac42ef9a20251007f39a56
I20251215 08:59:30.704813 21381 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.706192 21339 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
W20251215 08:59:30.708283 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:30.708029 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:30.710889 21307 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:30.714253 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:30.679852 23665 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:30.720836 23665 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:30.721536 21980 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:30.715075 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:30.718303 23705 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:30.696683 23621 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:30.702354 23403 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:30.730083 21974 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:30.710171 23646 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:30.711601 23655 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:30.823853 21412 raft_consensus.cc:1275] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251215 08:59:30.743409 23550 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:30.868039 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 163313us
W20251215 08:59:30.877022 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 166171us
I20251215 08:59:30.878060 21307 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:30.879635 23713 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:30.882148 21946 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:30.883169 23655 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:30.885867 21307 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.886538 23713 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:30.893152 23631 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:30.894483 23631 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:30.896108 23407 raft_consensus.cc:3055] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:30.896536 23407 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:30.897346 23407 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:30.898509 23407 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:30.906612 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:30.907102 21415 raft_consensus.cc:606] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:30.909924 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:30.915558 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:30.917953 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:30.922839 21411 raft_consensus.cc:1275] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:30.922955 21415 raft_consensus.cc:1275] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:30.886914 23713 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:30.931080 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:30.875494 23621 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:30.932482 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:31.017773 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:31.018810 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:31.019378 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:30.990137 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 109196us
I20251215 08:59:31.019464 21411 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
W20251215 08:59:31.000499 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:31.005923 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:31.012014 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:31.023085 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:31.023746 21411 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:30.993446 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:31.018478 21415 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:31.026051 21412 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:30.992028 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:31.026872 21259 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:31.018863 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:31.027961 21265 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:31.028122 21264 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:31.031019 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:30.908298 23651 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:30.983961 23682 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.034966 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:31.035974 21414 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:31.041636 21408 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:31.041705 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:31.025281 21263 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:31.037654 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.038286 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:31.038978 21411 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:31.043533 21411 raft_consensus.cc:606] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:31.043866 21411 raft_consensus.cc:614] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:31.050367 21307 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.055500 21232 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.056854 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:31.057523 21337 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:31.049501 21978 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20251215 08:59:31.066111 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 171055us
I20251215 08:59:31.072180 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.033581 23713 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:31.079522 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:30.924943 23550 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.064296 23655 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:30.984897 23684 consensus_queue.cc:1059] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20251215 08:59:30.930070 23544 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.143s
I20251215 08:59:31.069512 23670 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:31.049886 23618 consensus_queue.cc:1059] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.053s
I20251215 08:59:31.128997 21266 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:31.110715 21946 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:31.130455 21946 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:31.084949 23682 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:31.111639 23665 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:31.075430 23181 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:31.039366 23621 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.124917 21411 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:31.155946 21411 raft_consensus.cc:606] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:31.125154 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.141925 21266 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:31.109431 21337 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:31.074676 23407 consensus_queue.cc:1059] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20251215 08:59:31.150954 23665 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:31.165812 23665 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:31.114266 23653 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20251215 08:59:31.146143 23705 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:31.225353 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20251215 08:59:31.257206 21944 leader_election.cc:336] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.523s (SENT)
I20251215 08:59:31.132316 23565 consensus_queue.cc:1059] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20251215 08:59:31.156507 23711 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:31.258427 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:31.263334 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:31.154791 23626 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:31.155576 23655 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:31.161324 23670 raft_consensus.cc:3055] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:31.260308 23350 consensus_queue.cc:1048] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251215 08:59:31.261278 23716 log.cc:927] Time spent T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.221s user 0.003s sys 0.005s
I20251215 08:59:31.147538 23631 consensus_queue.cc:1059] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20251215 08:59:31.225244 21216 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 54676.4us
I20251215 08:59:31.173527 23407 consensus_queue.cc:1243] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer 195a56ac72a84018880bfb3bf6e6ef1d log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20251215 08:59:31.301939 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:31.260938 23706 log.cc:927] Time spent T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.098s user 0.004s sys 0.003s
I20251215 08:59:31.265815 23711 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:31.277091 23626 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:31.293761 23621 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:31.294356 23670 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:31.317878 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 200001us
W20251215 08:59:31.409361 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 274872us
W20251215 08:59:31.308971 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 178232us
I20251215 08:59:31.428025 21381 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:31.312675 23717 log.cc:927] Time spent T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.180s user 0.000s sys 0.000s
I20251215 08:59:31.301992 23705 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:31.445628 23725 log.cc:927] Time spent T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20251215 08:59:31.447132 23705 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:31.313251 23718 log.cc:927] Time spent T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20251215 08:59:31.401607 23711 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:31.312228 23631 raft_consensus.cc:3055] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:31.453969 23631 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:31.454804 23631 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.455120 23407 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:31.456043 23631 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:31.414945 23565 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:31.457525 23565 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.459614 23565 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:31.461807 23733 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:31.462491 23733 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.464766 23733 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:31.432088 23231 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20251215 08:59:31.431955 23181 consensus_queue.cc:1048] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20251215 08:59:31.409075 23626 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:31.473340 23670 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.498158 23670 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:31.506232 23231 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:31.506768 23231 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.509167 23231 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:31.530207 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 450777us
W20251215 08:59:31.530515 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 111744us
I20251215 08:59:31.533783 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.536996 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.538345 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.545627 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:31.546466 21261 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:31.543498 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:31.528254 23626 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.540958 23621 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:31.544332 23733 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:31.541806 23738 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20251215 08:59:31.568884 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 414642us
W20251215 08:59:31.580839 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:31.575286 21379 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.585772 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.587245 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.536677 23727 consensus_queue.cc:1059] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20251215 08:59:31.547580 23665 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.568352 23740 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:31.583600 23550 raft_consensus.cc:2804] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:31.608467 23665 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:31.608959 21262 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:31.592743 23711 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.621104 23587 raft_consensus.cc:3055] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:31.614418 23627 raft_consensus.cc:993] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:31.615052 21379 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.552066 23713 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.598415 21307 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.598615 23684 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:31.599320 23682 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.651795 21379 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:31.652987 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 55263.6us
I20251215 08:59:31.654666 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.623812 23550 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:31.663846 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 234348us
I20251215 08:59:31.622156 23736 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:31.667687 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.674501 21379 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:31.669531 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 239536us
I20251215 08:59:31.675732 21379 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.675792 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.670011 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.636672 23587 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:31.683874 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.671661 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.672431 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.685887 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.686093 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.677443 21232 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:31.687307 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.673370 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.673768 21973 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:31.674700 21333 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:31.670869 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.678623 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.689020 21337 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:31.689045 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.689832 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.689939 21333 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:31.690791 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.691174 21337 raft_consensus.cc:1275] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251215 08:59:31.691569 21337 pending_rounds.cc:85] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:31.691831 21337 pending_rounds.cc:107] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251215 08:59:31.692106 21337 raft_consensus.cc:2889] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:31.680694 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.688397 21332 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:31.664494 22925 raft_consensus.cc:1081] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:31.697681 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.698879 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:31.712015 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:31.623658 23740 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:31.673702 23550 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:31.706180 21381 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:31.718144 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1661) took 1121 ms. Trace:
I20251215 08:59:31.678475 23350 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.736631 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.729432 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.678788 23682 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.683490 23631 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.678292 23366 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:31.683200 23705 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:31.669198 23735 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.731544 21413 rpcz_store.cc:276] 1215 08:59:30.596356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:30.596617 (+ 261us) service_pool.cc:225] Handling call
1215 08:59:30.596923 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:30.597150 (+ 227us) raft_consensus.cc:1534] Early marking committed up to index 4
1215 08:59:30.822290 (+225140us) log.cc:844] Serialized 14 byte log entry
1215 08:59:31.319785 (+497495us) log.cc:844] Serialized 14 byte log entry
1215 08:59:31.476955 (+157170us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b48001bc258. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398cacde21 00007f398cac4eb7 00007f398cac4d4a 00007f398cbbbf7f 00007f398cbbbc47 00007f398cbd026c 00007f398cbd0218 00007f398cbd0198 00007f398cbd0153 00007f398cbcf3fc 00007f398bb0a516
1215 08:59:31.717507 (+240552us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:31.717576 (+ 69us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:31.717795 (+ 219us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:31.717914 (+ 119us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:31.718081 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16951,"prepare.queue_time_us":507475,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":166,"raft.queue_time_us":236292,"raft.run_cpu_time_us":2159,"raft.run_wall_time_us":46722,"spinlock_wait_cycles":63060352,"thread_start_us":1092659,"threads_started":3,"wal-append.queue_time_us":651411}
I20251215 08:59:31.678035 23747 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:31.703384 23231 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:31.740830 21381 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:31.761683 21944 leader_election.cc:336] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.175s (SENT)
I20251215 08:59:31.715493 23544 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:31.765375 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:31.762163 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 204776us
I20251215 08:59:31.782315 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.704267 23565 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.065s
I20251215 08:59:31.789677 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:31.797675 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 56932.8us
I20251215 08:59:31.744509 23736 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:31.802153 21334 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:31.802520 21230 leader_election.cc:336] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.737s (SENT)
I20251215 08:59:31.798686 21381 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:31.805441 21338 raft_consensus.cc:2410] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:31.802958 21230 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:31.817262 21230 leader_election.cc:336] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.811s (SENT)
I20251215 08:59:31.761988 23750 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:31.772138 23745 raft_consensus.cc:3055] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:31.779551 23350 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:31.818595 21381 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:31.771112 23705 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:31.818751 21230 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:31.824561 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:31.777788 23231 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:31.797322 23626 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:31.814476 23366 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:31.830735 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:31.777599 23617 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:31.836321 21339 raft_consensus.cc:2410] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251215 08:59:31.836966 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:31.819679 23627 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:31.818918 23665 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:31.838261 21974 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:31.822383 23544 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:31.849188 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:31.851469 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:31.854735 21979 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:31.823484 23745 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:31.852675 21976 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:31.832995 23705 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:31.832082 23350 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:31.853929 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:31.766894 23587 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.856868 21333 raft_consensus.cc:1275] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251215 08:59:31.884527 21333 pending_rounds.cc:85] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:31.884757 21333 pending_rounds.cc:107] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251215 08:59:31.884960 21333 raft_consensus.cc:2889] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:31.851976 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.833894 23231 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:31.881891 21340 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:31.841146 23366 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:31.885872 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 123142us
I20251215 08:59:31.851095 23627 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:31.889336 21379 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:31.888566 21333 raft_consensus.cc:1275] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251215 08:59:31.891405 21333 pending_rounds.cc:85] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:31.888196 23749 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:31.887494 23713 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:31.851017 23617 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:31.893988 23742 log.cc:927] Time spent T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20251215 08:59:31.894781 23617 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:31.890458 23366 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:31.895846 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1671) took 1332 ms. Trace:
W20251215 08:59:31.896166 23748 log.cc:927] Time spent T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251215 08:59:31.896225 21341 rpcz_store.cc:276] 1215 08:59:30.566254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:30.566387 (+ 133us) service_pool.cc:225] Handling call
1215 08:59:30.570199 (+ 3812us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:30.570553 (+ 354us) raft_consensus.cc:1534] Early marking committed up to index 7
1215 08:59:30.900709 (+330156us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b18004d7940. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b03045 00007f398b5ae317 00007f3987bbe627 00007f3987bcdd3f 00007f3987bcdcb8
1215 08:59:31.081928 (+181219us) log.cc:844] Serialized 14 byte log entry
1215 08:59:31.538905 (+456977us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:31.548095 (+ 9190us) log.cc:844] Serialized 26 byte log entry
1215 08:59:31.548303 (+ 208us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:31.548350 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:31.548435 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:31.895510 (+347075us) raft_consensus.cc:1685] finished
1215 08:59:31.895625 (+ 115us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:31.895790 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29899,"prepare.queue_time_us":302412,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":1186,"raft.queue_time_us":196,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":25767,"spinlock_wait_cycles":24234880,"thread_start_us":570159,"threads_started":2,"wal-append.queue_time_us":564810}
I20251215 08:59:31.891629 21333 pending_rounds.cc:107] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251215 08:59:31.900525 21333 raft_consensus.cc:2889] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:31.854475 23544 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:31.887240 23350 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:31.908613 23231 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.931157 21333 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251215 08:59:31.910181 23556 raft_consensus.cc:3055] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:31.881819 23741 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:31.912853 22925 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
W20251215 08:59:31.933084 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 43825.5us
I20251215 08:59:31.908376 23708 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:31.919023 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:31.939738 21262 raft_consensus.cc:1240] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20251215 08:59:31.824569 23750 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.901463 23627 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:31.958941 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:31.980738 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:32.004307 21258 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:32.004474 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:32.006968 21974 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:31.933749 23544 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:31.953495 21229 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:32.012467 21339 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20251215 08:59:32.012820 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 59430.3us
W20251215 08:59:32.009050 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 191788us
I20251215 08:59:31.940078 23741 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:32.020116 21230 leader_election.cc:336] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.842s (SENT)
I20251215 08:59:31.941534 23700 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:31.927048 23740 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:31.940517 23556 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:32.022038 21230 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:31.977998 23617 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:31.930068 23550 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.988916 23621 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:32.030118 21230 leader_election.cc:336] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.863s (SENT)
I20251215 08:59:32.033222 23708 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:31.979184 23684 consensus_queue.cc:1059] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20251215 08:59:32.059018 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:32.042070 23556 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.978497 23735 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:32.026582 23181 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:31.927932 23366 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:31.931798 23350 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.017618 23665 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:32.027860 23741 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:32.003118 23681 consensus_queue.cc:1059] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20251215 08:59:32.069792 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:32.070230 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:32.076750 21977 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:32.044263 23617 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:32.110733 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 90182.9us
I20251215 08:59:32.112067 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:32.121863 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:32.114269 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:32.116775 21944 leader_election.cc:400] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421) with higher term. Message: Invalid argument: T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:32.124874 21230 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:32.118613 21263 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:32.119021 21260 raft_consensus.cc:2393] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:32.119334 21261 raft_consensus.cc:1275] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251215 08:59:32.110589 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 225) took 1060 ms. Trace:
I20251215 08:59:32.127869 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:32.130190 21230 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:32.128584 21408 rpcz_store.cc:276] 1215 08:59:31.050682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:31.051580 (+ 898us) service_pool.cc:225] Handling call
1215 08:59:31.079102 (+ 27522us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:31.095298 (+ 16196us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:31.095343 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:31.103571 (+ 8228us) log.cc:844] Serialized 26 byte log entry
1215 08:59:31.321006 (+217435us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:31.346620 (+ 25614us) log.cc:844] Serialized 14 byte log entry
1215 08:59:31.915890 (+569270us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b18003e6380. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b03045 00007f3987babce0 00007f3987bbe396 00007f3987bcef07 00007f3987bcee88
1215 08:59:32.070930 (+155040us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:32.071240 (+ 310us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:32.071282 (+ 42us) raft_consensus.cc:1685] finished
1215 08:59:32.110374 (+ 39092us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:32.110530 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28380,"prepare.queue_time_us":25348,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":134803,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":14629,"spinlock_wait_cycles":35924864,"thread_start_us":372003,"threads_started":2,"wal-append.queue_time_us":445226}
I20251215 08:59:32.032656 23638 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20251215 08:59:32.113931 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:32.113595 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:32.136188 21260 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:32.136906 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:32.137791 21262 raft_consensus.cc:1240] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20251215 08:59:32.138172 21261 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251215 08:59:32.140988 21973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 598) took 1062 ms. Trace:
W20251215 08:59:32.174715 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.174903 21973 rpcz_store.cc:276] 1215 08:59:31.078197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:31.080935 (+ 2738us) service_pool.cc:225] Handling call
1215 08:59:31.112069 (+ 31134us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:31.688343 (+576274us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:32.140700 (+452357us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b540039c1d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:32.140920 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1291341568}
I20251215 08:59:32.075948 23632 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:32.076200 23708 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:32.154793 21413 raft_consensus.cc:1275] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251215 08:59:32.076107 23665 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:32.177603 23665 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:32.169955 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:32.076666 23735 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.109112 23617 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:32.195498 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 191900us
I20251215 08:59:32.170333 21337 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:32.100864 23670 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:32.170791 21338 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
W20251215 08:59:32.196055 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.202294 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:31.992225 23745 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.080298 23556 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:32.134264 23740 raft_consensus.cc:2749] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:32.152276 23757 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20251215 08:59:32.222437 23763 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:32.224234 23366 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:32.076318 23181 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:32.228241 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.101367 23682 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:32.143678 23565 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:32.233418 23627 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.276986 21408 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:32.277294 23735 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.277463 21973 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:32.279392 21412 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
W20251215 08:59:32.281961 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 79724.8us
I20251215 08:59:32.296419 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:32.270428 23766 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20251215 08:59:32.300563 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 170465us
I20251215 08:59:32.290274 21409 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:32.280330 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:32.307407 21230 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:32.307561 21414 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:32.310570 21230 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:32.311731 21230 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:32.315069 21230 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:32.233973 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 96257us
W20251215 08:59:32.316809 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.315136 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:32.275794 23618 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:32.276922 23750 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.274657 23350 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:32.283999 23627 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:32.289909 23181 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.341043 23781 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:32.353616 23781 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:32.354004 23781 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:32.340417 21230 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:32.341305 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 158539us
I20251215 08:59:32.282191 23747 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:32.282053 23745 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:32.300271 23565 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:32.287024 23768 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:32.293941 23550 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.348896 23618 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.313553 23740 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20251215 08:59:32.281488 23708 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20251215 08:59:32.343317 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.367511 21972 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:32.362525 23750 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:32.330880 23765 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:32.287842 23682 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.276149 23705 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.230326 23762 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:32.376572 21230 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:32.234448 23231 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.374753 23565 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:32.229962 23670 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.379033 23488 raft_consensus.cc:3055] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:32.379475 23768 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.298729 23754 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:32.340096 23785 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:32.342217 23778 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:32.440030 23488 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:32.384203 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 83247us
I20251215 08:59:32.449064 23488 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.450160 21379 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:32.393311 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 50098.8us
W20251215 08:59:32.450614 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 90047.7us
I20251215 08:59:32.392336 23635 consensus_queue.cc:1059] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20251215 08:59:32.394243 23704 consensus_queue.cc:1059] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20251215 08:59:32.395046 23735 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:32.395777 23711 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.396021 23708 raft_consensus.cc:3055] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:32.343250 23763 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.398744 23682 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.399974 23587 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:32.400465 23705 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:32.400933 23665 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.401516 23618 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.401743 21975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 599) took 1316 ms. Trace:
I20251215 08:59:32.455808 22927 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:32.474611 22927 raft_consensus.cc:697] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:32.473883 21975 rpcz_store.cc:276] 1215 08:59:31.085306 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:31.086852 (+ 1546us) service_pool.cc:225] Handling call
1215 08:59:31.131573 (+ 44721us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:31.590898 (+459325us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:31.593316 (+ 2418us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:31.601129 (+ 7813us) log.cc:844] Serialized 26 byte log entry
1215 08:59:31.813333 (+212204us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:31.819781 (+ 6448us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:31.825224 (+ 5443us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:32.391138 (+565914us) raft_consensus.cc:1685] finished
1215 08:59:32.391642 (+ 504us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:32.401674 (+ 10032us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35072,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":91622,"thread_start_us":626371,"threads_started":2,"wal-append.queue_time_us":259007}
I20251215 08:59:32.402217 23765 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:32.475764 22927 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.476928 23765 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.452459 23488 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:32.402393 23617 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.480692 23670 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.481900 23617 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.392673 23550 consensus_queue.cc:1059] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20251215 08:59:32.400000 23544 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.484038 23544 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:32.403442 23747 raft_consensus.cc:697] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:32.484634 23763 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.404493 21230 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:32.404995 23762 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:32.485527 23762 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:32.485610 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 81183.3us
I20251215 08:59:32.407167 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:32.486375 21972 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:32.486840 23768 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.378878 23774 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:32.487516 23774 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:32.410257 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.488924 21415 raft_consensus.cc:1240] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 2. Current term is 3. Ops: []
I20251215 08:59:32.416567 23733 raft_consensus.cc:3055] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:32.490051 23733 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:32.490206 23704 raft_consensus.cc:3055] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:32.490675 23704 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:32.490852 23733 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.492110 23733 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:32.491636 23704 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:32.417870 23736 consensus_meta.cc:287] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.594s user 0.003s sys 0.006s
I20251215 08:59:32.493264 23736 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:32.419044 23777 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:32.420504 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 540) took 1397 ms. Trace:
I20251215 08:59:32.494683 23785 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:32.493983 21971 rpcz_store.cc:276] 1215 08:59:31.022863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:31.992786 (+969923us) service_pool.cc:225] Handling call
1215 08:59:31.996864 (+ 4078us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:32.001537 (+ 4673us) raft_consensus.cc:1534] Early marking committed up to index 6
1215 08:59:32.040575 (+ 39038us) log.cc:844] Serialized 14 byte log entry
1215 08:59:32.353003 (+312428us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b18004e34c0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:32.419669 (+ 66666us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b18004e34c0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f398775b85c 00007f3984ae8ffb 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f39877535da
1215 08:59:32.420004 (+ 335us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:32.420093 (+ 89us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:32.420162 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:32.420275 (+ 113us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:32.420446 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":30008,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":95,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":45337088,"thread_start_us":59394,"threads_started":1,"wal-append.queue_time_us":59752}
I20251215 08:59:32.421667 23746 raft_consensus.cc:3055] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:32.425177 23707 raft_consensus.cc:993] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:32.497465 21304 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:32.498428 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:32.457260 23708 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:32.451964 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 90131.1us
I20251215 08:59:32.454726 23754 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.456444 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.435942 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:32.448040 23403 raft_consensus.cc:993] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower beefe15185ac42ef9a20251007f39a56 to start an election
I20251215 08:59:32.510079 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:32.465085 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.447711 23743 raft_consensus.cc:3055] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:32.481775 23565 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.482316 23665 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.494771 23777 consensus_queue.cc:1048] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:32.416392 23781 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.505188 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:32.509191 21262 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:32.492990 23704 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:32.458241 23741 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.495898 23746 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:32.499218 23707 raft_consensus.cc:1081] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
W20251215 08:59:32.234745 23713 consensus_queue.cc:1175] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" responder_term: 8 status { last_received { term: 6 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 7. Current term is 8. Ops: [7.8-7.8]" } } last_received_current_leader { term: 0 index: 0 } }
W20251215 08:59:32.527109 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 69525.6us
I20251215 08:59:32.510421 23653 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:32.534346 23556 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:32.538538 23565 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.512789 23743 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:32.540395 23781 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.531143 23544 raft_consensus.cc:1081] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Signalling peer beefe15185ac42ef9a20251007f39a56 to start an election
I20251215 08:59:32.532222 23714 raft_consensus.cc:3055] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:32.542339 23714 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:32.493697 23747 consensus_queue.cc:237] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.543884 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:32.539327 23556 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.545079 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:32.545820 21260 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:32.549702 21261 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:32.549530 23743 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.544670 23708 consensus_queue.cc:260] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.551874 23774 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.543171 23714 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.554050 23556 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:32.552429 23762 consensus_queue.cc:1175] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" responder_term: 3 status { last_received { term: 3 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 3 index: 5 } }
I20251215 08:59:32.556542 21978 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:32.557843 23653 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.560067 23653 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.561481 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:32.565457 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:32.566078 21262 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:32.566676 23708 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:32.568231 23743 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:32.569180 23714 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:32.571471 23754 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.529337 23741 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:32.574805 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:32.576201 23741 consensus_queue.cc:1059] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:32.511397 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:32.587899 21971 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:32.588092 23670 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:32.588577 23670 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:32.588938 23670 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:32.589490 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:32.590119 21263 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:32.587889 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 90501.6us
I20251215 08:59:32.592908 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:32.593350 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 136984us
I20251215 08:59:32.593816 21261 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:32.594508 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.595028 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:32.601644 21340 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:32.602190 21340 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:32.602522 21340 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:32.602556 21337 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:32.603837 21336 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251215 08:59:32.604184 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:32.603879 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:32.605683 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:32.606670 21338 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251215 08:59:32.607935 21333 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251215 08:59:32.613616 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.614724 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.615165 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:32.617331 21260 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba"
dest_uuid: "beefe15185ac42ef9a20251007f39a56"
from {username='slave'} at 127.0.0.1:50710
I20251215 08:59:32.618053 21260 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:32.618489 21260 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:32.621963 21307 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.623009 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.623992 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.624442 21262 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:32.624550 21263 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:32.626569 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:32.645572 21258 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:32.647372 21264 raft_consensus.cc:1240] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:32.654654 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.655789 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:32.612244 23618 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:32.673355 21261 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:32.673488 21260 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:32.677721 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.654174 23774 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:32.681811 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:32.680691 21340 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:32.661141 23781 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:32.693394 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:32.684396 23684 consensus_queue.cc:1059] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20251215 08:59:32.661695 23682 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:32.690096 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:32.698083 21411 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:32.672295 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:32.690435 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:32.662533 23488 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:32.663648 23783 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:32.662251 23681 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:32.662851 23550 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:32.683881 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:32.699343 21260 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:32.646831 23403 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20251215 08:59:32.692533 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:32.731216 21410 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:59:32.703140 21162 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45538 (request call id 151) took 11670 ms (client timeout 15000 ms). Trace:
I20251215 08:59:32.703086 21381 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:32.702371 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:32.732496 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:32.732796 21978 raft_consensus.cc:2393] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:32.705391 21409 raft_consensus.cc:1240] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:32.700973 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:32.697122 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:32.673943 23780 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:32.699486 21340 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:32.677522 23779 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:32.697758 23735 consensus_queue.cc:1059] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20251215 08:59:32.711447 21163 catalog_manager.cc:5654] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 5, leader changed from f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:32.716266 21411 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:32.716660 23774 consensus_queue.cc:1059] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:32.697510 23618 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:32.591228 23746 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.728482 23670 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:32.681347 23775 consensus_queue.cc:1048] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
W20251215 08:59:33.112702 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:32.802181 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 99171.7us
W20251215 08:59:33.191044 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.302879 21381 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:32.910952 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:32.883180 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 189282us
W20251215 08:59:33.143257 23714 consensus_meta.cc:287] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.574s user 0.009s sys 0.000s
I20251215 08:59:33.306565 21337 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20251215 08:59:33.306941 21335 pending_rounds.cc:85] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 5
I20251215 08:59:33.307252 21335 pending_rounds.cc:107] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20251215 08:59:33.307516 21335 raft_consensus.cc:2889] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:33.306802 21413 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:32.801496 21333 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
W20251215 08:59:33.309330 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:32.699878 23781 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.160655 21415 raft_consensus.cc:1217] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Deduplicated request from leader. Original: 2.2->[4.3-4.3] Dedup: 4.3->[]
I20251215 08:59:33.312036 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.312474 21415 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251215 08:59:33.313555 21336 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:33.312417 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.313885 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.163053 23682 raft_consensus.cc:3055] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:33.314734 21339 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:33.314910 23682 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:33.316195 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.316525 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.316288 23682 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.317446 21339 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:32.819742 23791 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20251215 08:59:33.319773 23670 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.320479 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.321182 21410 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:33.321990 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:33.324168 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.325839 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.327035 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.327080 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:33.327754 21413 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:33.328624 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.331094 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:33.329619 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:33.333066 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 598898us
I20251215 08:59:32.798703 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:33.340903 21408 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:32.808574 21338 raft_consensus.cc:1240] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:32.797051 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.323016 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:33.350644 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.350731 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.352252 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.355261 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.359833 21406 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:33.357669 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.362746 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.365238 21338 raft_consensus.cc:2410] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:33.364001 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.363517 21163 catalog_manager.cc:5654] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 3, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:33.367159 23682 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:32.811038 23014 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:33.367820 21163 catalog_manager.cc:5654] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 4, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 4 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:33.370002 21163 catalog_manager.cc:5654] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:33.358352 23784 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.371300 23784 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.372661 21337 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:33.373340 23784 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.374004 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:32.960564 23783 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:33.375615 23783 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.376029 23684 raft_consensus.cc:3055] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:33.376489 23684 raft_consensus.cc:740] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:33.135280 23488 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.377337 23488 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.377696 23783 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.306730 23780 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:33.378322 23780 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:33.377334 23684 consensus_queue.cc:260] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.379974 23762 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.380406 23684 raft_consensus.cc:3060] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:33.380687 23746 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:33.380489 23762 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.381960 23781 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.338343 23618 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:33.382699 23762 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.382931 23618 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:33.363662 23779 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.384557 23762 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.384624 23779 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.385146 23762 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.386934 23779 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.387950 21409 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
W20251215 08:59:33.389569 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 86808.9us
W20251215 08:59:32.731612 21162 rpcz_store.cc:269] 1215 08:59:21.018353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:21.018559 (+ 206us) service_pool.cc:225] Handling call
1215 08:59:32.688538 (+11669979us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:29.967227 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:29.967816 (+ 589us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:29.967858 (+ 42us) write_op.cc:435] Acquired schema lock
1215 08:59:29.967878 (+ 20us) tablet.cc:660] Decoding operations
1215 08:59:29.968944 (+ 1066us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:29.969031 (+ 87us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:29.969064 (+ 33us) tablet.cc:683] Acquiring locks for 9 operations
1215 08:59:29.969835 (+ 771us) tablet.cc:699] Row locks acquired
1215 08:59:29.969864 (+ 29us) write_op.cc:260] PREPARE: finished
1215 08:59:29.970020 (+ 156us) write_op.cc:270] Start()
1215 08:59:29.970164 (+ 144us) write_op.cc:276] Timestamp: P: 1765789169969965 usec, L: 0
1215 08:59:29.970188 (+ 24us) op_driver.cc:348] REPLICATION: starting
1215 08:59:29.970921 (+ 733us) log.cc:844] Serialized 3601 byte log entry
1215 08:59:30.738444 (+767523us) op_driver.cc:464] REPLICATION: finished
1215 08:59:31.018958 (+280514us) write_op.cc:301] APPLY: starting
1215 08:59:31.033097 (+ 14139us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:31.081659 (+ 48562us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:31.129166 (+ 47507us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:31.457485 (+328319us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:31.457715 (+ 230us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
1215 08:59:31.457759 (+ 44us) write_op.cc:312] APPLY: finished
1215 08:59:31.458031 (+ 272us) log.cc:844] Serialized 71 byte log entry
1215 08:59:31.458240 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:31.458371 (+ 131us) write_op.cc:454] Released schema lock
1215 08:59:31.458419 (+ 48us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:31.458790 (+ 371us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:31.458820 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14689,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":185988,"apply.run_cpu_time_us":4435,"apply.run_wall_time_us":464901,"num_ops":9,"prepare.queue_time_us":281028,"prepare.run_cpu_time_us":27547,"prepare.run_wall_time_us":254915,"raft.queue_time_us":190508,"raft.run_cpu_time_us":9918,"raft.run_wall_time_us":121248,"replication_time_us":745080,"thread_start_us":675404,"threads_started":4,"wal-append.queue_time_us":342259,"wal-append.run_cpu_time_us":70870,"wal-append.run_wall_time_us":2147649}]]}
I20251215 08:59:33.390331 21381 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:33.389662 23014 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:33.391655 21341 raft_consensus.cc:2376] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7. Re-sending same reply.
I20251215 08:59:33.274384 23736 raft_consensus.cc:3055] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:33.395397 21408 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:33.376231 21163 catalog_manager.cc:5654] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 4, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 4 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:33.391134 21381 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:33.401106 22925 raft_consensus.cc:2804] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:33.410463 23774 raft_consensus.cc:3055] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Stepping down as leader of term 2
I20251215 08:59:32.770200 21975 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:33.414335 23775 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:33.417204 23751 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:33.417816 23751 raft_consensus.cc:697] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:33.417579 23775 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.418744 23751 consensus_queue.cc:237] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.419739 23775 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.421151 22925 raft_consensus.cc:697] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:33.422227 22925 consensus_queue.cc:237] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.425794 21381 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:33.421216 23488 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.444576 21259 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:33.411708 23762 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.404361 23736 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:33.433117 21971 consensus_meta.cc:287] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.826s user 0.009s sys 0.001s
I20251215 08:59:33.457864 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.414404 23774 raft_consensus.cc:740] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:33.460872 21944 leader_election.cc:336] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.511s (SENT)
I20251215 08:59:33.461378 21944 leader_election.cc:304] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:33.462806 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.463515 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:33.444633 23635 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:33.465953 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.466528 21340 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:33.466594 21230 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:33.462635 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.445127 23781 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:33.470320 21230 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:33.468698 21337 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:33.474483 23403 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.475028 23403 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.469784 23777 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:33.467895 23762 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:33.479609 23750 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:33.479838 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:33.480015 23750 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:33.480366 23750 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:33.480409 21978 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:33.469695 23488 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.481550 21378 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:33.482518 23350 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:33.482160 23488 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.482884 23350 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:33.483232 23350 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:33.484429 23488 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.485096 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:33.486889 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.487577 21409 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:33.488979 21340 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:33.489542 21340 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:33.489593 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.489957 21340 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:33.444728 23765 consensus_queue.cc:1059] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20251215 08:59:33.493530 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:33.465351 23708 consensus_meta.cc:287] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.898s user 0.010s sys 0.001s
I20251215 08:59:33.456722 21971 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:33.476833 23403 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.464459 23784 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251215 08:59:33.504673 21412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.024s sys 0.033s
I20251215 08:59:33.500972 21978 raft_consensus.cc:3055] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:33.503477 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:33.506603 21412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.025s sys 0.033s
I20251215 08:59:33.506770 21978 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:33.490574 23795 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:33.524689 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.511790 21978 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.536542 21978 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:33.537981 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.542310 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.546496 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:33.557438 21944 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:33.560779 21944 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:33.494617 23781 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.567122 21332 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:33.569913 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.510987 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 610) took 1008 ms. Trace:
I20251215 08:59:33.577723 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:33.572908 21944 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:33.502566 23736 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.588243 21944 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:33.588299 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.589412 21944 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:33.541838 23708 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.537232 23746 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:33.604434 23746 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.607327 23746 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.592952 23682 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:33.601511 21264 raft_consensus.cc:1240] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:33.587253 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.609638 21260 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:33.611523 21409 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20251215 08:59:33.620031 21407 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.014s sys 0.026s
W20251215 08:59:33.622635 21407 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s user 0.015s sys 0.026s
I20251215 08:59:33.620759 23769 consensus_queue.cc:1059] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20251215 08:59:33.629062 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.575548 23777 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:33.622092 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.569900 23762 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:33.536216 23795 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.627279 23746 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.575238 23743 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:33.571265 23784 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:33.611567 21340 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.607928 23682 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.632874 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.592478 23654 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:33.651073 21339 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:33.642485 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.620879 21978 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:33.510403 23774 consensus_queue.cc:260] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.645207 23796 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.575657 21971 rpcz_store.cc:276] 1215 08:59:32.497615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:32.499459 (+ 1844us) service_pool.cc:225] Handling call
1215 08:59:33.506407 (+1006948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:33.569698 23780 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.660566 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.661636 21341 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:33.648587 21414 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.010s sys 0.038s
W20251215 08:59:33.680989 21414 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s user 0.010s sys 0.039s
I20251215 08:59:33.662896 21340 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.618598 23708 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:33.677714 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 59801.6us
I20251215 08:59:33.665542 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1700) took 1064 ms. Trace:
I20251215 08:59:33.682370 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.649575 23765 consensus_queue.cc:1059] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20251215 08:59:33.666278 23488 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:33.695971 23750 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.667723 23366 raft_consensus.cc:3055] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:33.669863 21415 raft_consensus.cc:3055] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:33.697923 21415 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:33.655540 23796 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.670154 23682 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.656253 23746 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.660458 23784 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:33.683516 21379 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:33.682766 21334 rpcz_store.cc:276] 1215 08:59:32.600702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:32.601108 (+ 406us) service_pool.cc:225] Handling call
1215 08:59:32.601487 (+ 379us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:32.613183 (+ 11696us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:32.613210 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:32.647018 (+ 33808us) log.cc:844] Serialized 26 byte log entry
1215 08:59:32.847452 (+200434us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:32.847511 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:32.847601 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:33.665277 (+817676us) raft_consensus.cc:1685] finished
1215 08:59:33.665306 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:33.665475 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20214,"raft.queue_time_us":21837,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":35556,"thread_start_us":166053,"threads_started":1,"wal-append.queue_time_us":955764}
I20251215 08:59:33.683437 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.694994 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.647629 23618 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.649787 23762 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.647086 23777 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.703451 21332 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:33.703406 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.708245 21334 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:33.708513 21266 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:33.709903 21334 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251215 08:59:33.698885 21415 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.677160 23654 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:33.695957 23765 raft_consensus.cc:3055] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:33.715041 21415 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:33.702188 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.699755 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.700585 23366 raft_consensus.cc:740] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:33.709782 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:33.681802 23774 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
W20251215 08:59:33.722383 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.722453 21337 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:33.700001 23743 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:33.719146 21339 raft_consensus.cc:1240] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251215 08:59:33.724418 21339 raft_consensus.cc:1240] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:33.705822 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:33.707221 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:33.724538 21340 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:33.707171 23544 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:33.696044 23768 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20251215 08:59:33.722680 23587 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.714493 23765 raft_consensus.cc:740] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:33.746831 23587 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.747136 23350 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:33.755262 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:33.710583 23705 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20251215 08:59:33.755872 21260 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:33.721822 23654 raft_consensus.cc:697] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:33.748787 23587 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.771225 21340 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
W20251215 08:59:33.781957 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 418613us
I20251215 08:59:33.738627 23747 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251215 08:59:33.748968 23350 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.787493 21266 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6. Ops: []
W20251215 08:59:33.760078 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 372962us
I20251215 08:59:33.752188 23777 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.766999 23781 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:33.861366 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.882882 21415 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:33.749981 23746 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.809578 21304 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:33.800697 21260 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
W20251215 08:59:33.887459 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 78009.8us
I20251215 08:59:33.783702 23788 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:33.751004 23618 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:33.805982 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:33.858621 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.896970 21266 raft_consensus.cc:1240] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:33.898898 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.899839 21406 raft_consensus.cc:2393] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 8: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:33.897001 21229 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:33.877537 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 150282us
I20251215 08:59:33.883890 21230 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:33.788447 23750 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:33.888074 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:33.889532 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:33.915683 21265 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:33.889526 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:33.916482 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.902604 21379 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:33.896008 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:33.781719 23769 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20251215 08:59:33.918541 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:33.919075 21406 raft_consensus.cc:3055] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:33.919379 21406 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:33.920482 21406 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:33.920995 21230 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:33.924130 21415 raft_consensus.cc:606] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Received request to transfer leadership to TS f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:33.870071 23784 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:33.925346 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.924419 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.921906 21406 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:33.925379 21415 raft_consensus.cc:614] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:33.919353 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.924177 21230 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:33.927359 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.928642 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:33.950536 21258 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:33.894799 23774 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:33.932622 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.972887 21409 raft_consensus.cc:2410] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20251215 08:59:33.933400 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1688) took 1244 ms. Trace:
I20251215 08:59:33.973259 21414 rpcz_store.cc:276] 1215 08:59:32.688773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:32.688890 (+ 117us) service_pool.cc:225] Handling call
1215 08:59:32.689311 (+ 421us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:32.696001 (+ 6690us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:32.696043 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:32.960279 (+264236us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b2c00092518. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:32.960783 (+ 504us) log.cc:844] Serialized 26 byte log entry
1215 08:59:33.681453 (+720670us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:33.681490 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:33.681565 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:33.933080 (+251515us) raft_consensus.cc:1685] finished
1215 08:59:33.933111 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:33.933342 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89562,"raft.queue_time_us":6711,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":7447,"spinlock_wait_cycles":5571456,"thread_start_us":710410,"threads_started":1,"wal-append.queue_time_us":920005}
I20251215 08:59:33.886204 23780 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.888767 23795 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.934127 23762 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:33.895171 23682 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:33.937229 23781 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:33.946228 23556 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election won for term 8
W20251215 08:59:33.941576 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:33.976342 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.981600 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:33.987011 21406 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:33.992241 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.857357 23765 consensus_queue.cc:260] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:33.937628 23798 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20251215 08:59:34.024050 21258 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:33.887317 23747 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20251215 08:59:34.032832 21259 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251215 08:59:33.979805 23624 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:34.036981 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:33.914096 23788 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.039328 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:34.050462 21261 raft_consensus.cc:2410] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:34.002856 23768 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:34.002979 23682 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:33.987654 23708 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:34.008241 23635 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.022s sys 0.022s
I20251215 08:59:33.992882 23750 consensus_queue.cc:1059] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20251215 08:59:34.034915 23014 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.023s sys 0.027s
I20251215 08:59:34.008585 23781 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:34.059183 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 469800us
I20251215 08:59:33.979455 23784 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.002710 23587 consensus_queue.cc:1059] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20251215 08:59:33.989235 23774 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.047909 23556 raft_consensus.cc:697] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:33.919950 23654 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.074734 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:34.055949 23796 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.058720 23624 raft_consensus.cc:697] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:34.073019 23014 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.594s user 0.024s sys 0.027s
W20251215 08:59:34.066941 22925 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.006s sys 0.014s
I20251215 08:59:34.080638 23708 raft_consensus.cc:3055] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:34.083302 23587 raft_consensus.cc:3055] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
W20251215 08:59:34.067713 23635 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s user 0.022s sys 0.022s
I20251215 08:59:34.066716 23682 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:34.091619 21944 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:34.091876 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:34.100217 21978 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:34.075856 23781 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.093511 23745 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:34.079592 23786 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20251215 08:59:34.095413 23708 raft_consensus.cc:740] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:34.098352 23587 raft_consensus.cc:740] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:34.100035 23635 raft_consensus.cc:697] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:34.096719 23624 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.105532 23801 raft_consensus.cc:2764] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
W20251215 08:59:34.093732 22925 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s user 0.010s sys 0.014s
W20251215 08:59:34.066637 23751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.009s sys 0.031s
I20251215 08:59:34.110090 23745 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.142957 23797 log.cc:927] Time spent T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.283s user 0.002s sys 0.000s
I20251215 08:59:34.119426 23556 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.181094 23804 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20251215 08:59:34.182487 23736 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:34.185498 23366 consensus_queue.cc:260] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.187268 23366 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:34.194495 23784 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.200579 23774 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.205291 23788 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.207306 23745 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.209218 23635 consensus_queue.cc:237] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.211407 23765 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:34.149688 23587 consensus_queue.cc:260] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.229732 23587 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:34.140143 23708 consensus_queue.cc:260] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.237761 23708 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:34.155346 23751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s user 0.010s sys 0.031s
I20251215 08:59:34.291796 23682 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.294984 23781 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.297721 23781 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:34.352080 21163 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.011s sys 0.028s
W20251215 08:59:34.353205 21163 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.890s user 0.011s sys 0.029s
W20251215 08:59:34.356652 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:34.053260 21232 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:34.367871 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 314559us
I20251215 08:59:34.369743 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:34.373070 21232 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:34.378959 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:34.381265 21260 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:34.382927 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:34.388407 21263 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.954s user 0.025s sys 0.030s
W20251215 08:59:34.425853 21335 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.001s sys 0.006s
W20251215 08:59:34.427124 21332 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s user 0.004s sys 0.022s
W20251215 08:59:34.427752 21334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.005s sys 0.021s
W20251215 08:59:34.460028 21334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s user 0.006s sys 0.022s
W20251215 08:59:34.039827 21336 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.011s sys 0.027s
W20251215 08:59:34.427963 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:34.040863 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:34.439141 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1689) took 1749 ms. Trace:
I20251215 08:59:34.441717 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1698) took 1745 ms. Trace:
I20251215 08:59:34.441784 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1694) took 1749 ms. Trace:
W20251215 08:59:34.452509 21946 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.000s sys 0.026s
I20251215 08:59:34.133265 21978 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:34.455014 23618 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:34.455264 23014 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:34.456748 23796 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:34.469573 21333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.004s sys 0.002s
W20251215 08:59:34.474105 21332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s user 0.005s sys 0.023s
W20251215 08:59:34.472339 21335 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.618s user 0.002s sys 0.007s
W20251215 08:59:34.475889 21263 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.041s user 0.026s sys 0.030s
W20251215 08:59:34.486668 21265 consensus_meta.cc:287] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.536s user 0.007s sys 0.007s
I20251215 08:59:34.486716 23750 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:34.494091 21379 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s user 0.002s sys 0.006s
W20251215 08:59:34.495728 21381 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.003s sys 0.006s
I20251215 08:59:34.128804 21971 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20251215 08:59:34.499991 21946 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.630s user 0.001s sys 0.026s
W20251215 08:59:34.501922 21381 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.003s sys 0.006s
W20251215 08:59:34.503805 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:34.503933 23762 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:34.504171 21263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1855) took 1072 ms. Trace:
I20251215 08:59:34.505025 23762 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:34.505885 23762 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:34.506501 23762 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.507002 23800 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:34.508106 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 248) took 1201 ms. Trace:
I20251215 08:59:34.508204 23708 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:34.507949 23618 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.508067 21336 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.005s user 0.012s sys 0.027s
I20251215 08:59:34.509382 23708 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.510924 21265 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
W20251215 08:59:34.502691 21333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.004s sys 0.003s
I20251215 08:59:34.500039 23745 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:34.512771 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 643252us
I20251215 08:59:34.513509 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1716) took 1020 ms. Trace:
I20251215 08:59:34.513418 23745 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.514420 23781 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:34.506049 21408 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:34.496824 21411 rpcz_store.cc:276] 1215 08:59:32.695819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.332300 (+636481us) service_pool.cc:225] Handling call
1215 08:59:33.332621 (+ 321us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:33.387342 (+ 54721us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:33.387373 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:33.387871 (+ 498us) log.cc:844] Serialized 26 byte log entry
1215 08:59:33.866061 (+478190us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:33.866105 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:33.866173 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:34.441202 (+575029us) raft_consensus.cc:1685] finished
1215 08:59:34.441252 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.441551 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54279,"raft.queue_time_us":55435,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":477782,"threads_started":1,"wal-append.queue_time_us":705715}
I20251215 08:59:34.515316 23796 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.515933 23745 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.504218 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:34.526371 23774 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251215 08:59:34.525777 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 416603us
W20251215 08:59:34.529201 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:34.529038 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:34.531030 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:34.531675 21944 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:34.533514 23784 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:34.534111 23784 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:34.534682 23784 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.538723 23743 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:34.539587 23682 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.541476 23801 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:34.541316 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.541981 23801 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:34.500175 23665 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:34.542474 23801 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:34.543067 23665 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.542490 21973 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:34.547623 23665 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.547617 23014 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.551029 23014 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:34.552070 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1451) took 1057 ms. Trace:
I20251215 08:59:34.544368 23800 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:34.496709 21412 rpcz_store.cc:276] 1215 08:59:32.689067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:32.689690 (+ 623us) service_pool.cc:225] Handling call
1215 08:59:32.689987 (+ 297us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:32.696034 (+ 6047us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:32.696061 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:32.887675 (+191614us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b2c00092518. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:32.888046 (+ 371us) log.cc:844] Serialized 26 byte log entry
1215 08:59:33.162335 (+274289us) spinlock_profiling.cc:243] Waited 4.65 ms on lock 0x7b48001bc258. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398cacde21 00007f398cac4eb7 00007f398cac4d4a 00007f398cbbbf7f 00007f398cbbbc47 00007f398cbd026c 00007f398cbd0218 00007f398cbd0198 00007f398cbd0153 00007f398cbcf3fc 00007f398bb0a516
1215 08:59:33.506994 (+344659us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:33.507036 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:33.507121 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:34.142622 (+635501us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b240011f678. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f398495a9a4 00007f3984958aab 00007f39849e79a9 00007f39849e7635 00007f39849e6c7d 00007f39877374ae 00007f398774ec2c 00007f398774df56 00007f398774db78 00007f39877575c1
1215 08:59:34.438345 (+295723us) raft_consensus.cc:1685] finished
1215 08:59:34.438411 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.438798 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":200464,"raft.queue_time_us":6164,"raft.run_cpu_time_us":2249,"raft.run_wall_time_us":464710,"spinlock_wait_cycles":51133952,"thread_start_us":618592,"threads_started":1,"wal-append.queue_time_us":865608}
W20251215 08:59:34.557454 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:34.508698 21335 rpcz_store.cc:276] 1215 08:59:33.306165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.306449 (+ 284us) service_pool.cc:225] Handling call
1215 08:59:33.306759 (+ 310us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:33.358609 (+ 51850us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:33.358642 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:33.359205 (+ 563us) log.cc:844] Serialized 52 byte log entry
1215 08:59:33.785957 (+426752us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:34.506246 (+720289us) log.cc:844] Serialized 14 byte log entry
1215 08:59:34.507269 (+ 1023us) log.cc:844] Serialized 14 byte log entry
1215 08:59:34.507533 (+ 264us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:34.507653 (+ 120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:34.507743 (+ 90us) raft_consensus.cc:1685] finished
1215 08:59:34.507759 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.507994 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112157,"prepare.queue_time_us":323591,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":2552,"raft.queue_time_us":59753,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":24742,"spinlock_wait_cycles":10496,"thread_start_us":1065805,"threads_started":2,"wal-append.queue_time_us":716791}
I20251215 08:59:34.521953 21410 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:34.496906 21407 rpcz_store.cc:276] 1215 08:59:32.691913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:32.692558 (+ 645us) service_pool.cc:225] Handling call
1215 08:59:32.692843 (+ 285us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:32.814281 (+121438us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:32.814326 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:32.888719 (+ 74393us) log.cc:844] Serialized 52 byte log entry
1215 08:59:33.624040 (+735321us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:34.069479 (+445439us) log.cc:844] Serialized 14 byte log entry
1215 08:59:34.069732 (+ 253us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:34.069846 (+ 114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:34.441319 (+371473us) raft_consensus.cc:1685] finished
1215 08:59:34.441358 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.441622 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":665142,"prepare.queue_time_us":514860,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":119323,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":11181,"thread_start_us":922048,"threads_started":2,"wal-append.queue_time_us":774225}
W20251215 08:59:34.557349 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 654821us
I20251215 08:59:34.552632 21408 rpcz_store.cc:276] 1215 08:59:33.494993 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.495141 (+ 148us) service_pool.cc:225] Handling call
1215 08:59:33.495720 (+ 579us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:34.515738 (+1020018us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:34.551693 (+ 35955us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400340b54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:34.551991 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2322380672}
I20251215 08:59:34.568804 23825 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
W20251215 08:59:34.586354 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 58649.8us
I20251215 08:59:34.582161 23800 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.574384 23796 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.504756 21263 rpcz_store.cc:276] 1215 08:59:33.431381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.433162 (+ 1781us) service_pool.cc:225] Handling call
1215 08:59:33.433515 (+ 353us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:33.433915 (+ 400us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:33.433951 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:33.434445 (+ 494us) log.cc:844] Serialized 26 byte log entry
1215 08:59:34.503492 (+1069047us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:34.503597 (+ 105us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:34.503715 (+ 118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:34.503792 (+ 77us) raft_consensus.cc:1685] finished
1215 08:59:34.503806 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.504015 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3437,"raft.queue_time_us":98,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":7323,"thread_start_us":1065072,"threads_started":1,"wal-append.queue_time_us":730003}
I20251215 08:59:34.577323 23618 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:34.597496 21410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1703) took 1892 ms (client timeout 1894 ms). Trace:
I20251215 08:59:34.514089 21333 rpcz_store.cc:276] 1215 08:59:33.493148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.493291 (+ 143us) service_pool.cc:225] Handling call
1215 08:59:33.495652 (+ 2361us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:33.495831 (+ 179us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:33.740715 (+244884us) log.cc:844] Serialized 14 byte log entry
1215 08:59:34.512775 (+772060us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:34.512882 (+ 107us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:34.513055 (+ 173us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:34.513183 (+ 128us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.513433 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":599384,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":1769,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":149,"thread_start_us":1008017,"threads_started":2,"wal-append.queue_time_us":729002}
I20251215 08:59:34.558733 23708 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:34.606016 21410 rpcz_store.cc:269] 1215 08:59:32.701473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.322795 (+621322us) service_pool.cc:225] Handling call
1215 08:59:34.593848 (+1271053us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540026cad4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:34.594159 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2498134400}
I20251215 08:59:34.579258 23682 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:34.608697 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:34.609555 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1867) took 1026 ms. Trace:
W20251215 08:59:34.602180 21231 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.024s sys 0.052s
W20251215 08:59:34.612318 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 233637us
I20251215 08:59:34.609551 23777 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:34.614988 21231 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.025s sys 0.052s
I20251215 08:59:34.613886 21262 rpcz_store.cc:276] 1215 08:59:33.583317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.584211 (+ 894us) service_pool.cc:225] Handling call
1215 08:59:33.584527 (+ 316us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:33.584756 (+ 229us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:34.052378 (+467622us) log.cc:844] Serialized 14 byte log entry
1215 08:59:34.556236 (+503858us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b1800464740. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:34.608893 (+ 52657us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:34.608983 (+ 90us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:34.609086 (+ 103us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:34.609178 (+ 92us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.609347 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":540980,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2608,"raft.queue_time_us":5013,"spinlock_wait_cycles":26197248,"thread_start_us":893231,"threads_started":2,"wal-append.queue_time_us":480764}
I20251215 08:59:34.620395 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1715) took 1128 ms. Trace:
W20251215 08:59:34.623342 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 90363.7us
I20251215 08:59:34.633045 21407 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:34.633988 21407 raft_consensus.cc:606] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:34.634411 21407 raft_consensus.cc:614] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:34.634898 23750 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:34.620832 23796 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:34.622790 21336 rpcz_store.cc:276] 1215 08:59:33.491774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.491899 (+ 125us) service_pool.cc:225] Handling call
1215 08:59:33.492141 (+ 242us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:33.492330 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 8
1215 08:59:34.510394 (+1018064us) log.cc:844] Serialized 14 byte log entry
1215 08:59:34.615555 (+105161us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:34.619731 (+ 4176us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:34.619880 (+ 149us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:34.620025 (+ 145us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:34.620320 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":583319,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":2228,"raft.queue_time_us":6444,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":6402,"thread_start_us":1110721,"threads_started":2}
I20251215 08:59:34.615890 23682 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.640760 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:34.636231 23750 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.644183 21407 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b83855234bda47a1b06c3f932728766b"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:34.631594 23618 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:34.646548 23774 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:34.648272 21407 raft_consensus.cc:606] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:34.648615 21407 raft_consensus.cc:614] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:34.652447 23777 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:34.657451 23777 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:34.662350 21407 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:34.663093 21407 raft_consensus.cc:606] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:34.666489 23750 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.667547 23837 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:34.668475 23827 log.cc:927] Time spent T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.092s user 0.000s sys 0.002s
I20251215 08:59:34.669667 21407 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:34.673979 23682 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:34.674865 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 47133.4us
I20251215 08:59:34.670534 21407 raft_consensus.cc:606] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:34.679755 21407 raft_consensus.cc:614] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:34.682651 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:59:34.689320 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:34.690873 23813 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:34.692917 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:34.694341 23813 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.695418 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:34.671423 23618 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.676482 23800 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:34.708333 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54893.5us
I20251215 08:59:34.711892 23801 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.720091 21232 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:34.668478 23837 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.722183 21379 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:34.725421 23635 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:34.725641 23700 consensus_queue.cc:1059] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:34.726150 23825 consensus_queue.cc:1059] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:34.727108 23014 raft_consensus.cc:3055] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:34.727519 23014 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:34.727636 23700 raft_consensus.cc:3055] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:34.728184 23700 raft_consensus.cc:740] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:34.728718 23014 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.732795 23700 consensus_queue.cc:260] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.734459 23014 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:34.736647 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:34.734517 23700 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:34.743783 23784 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.753249 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 60601.6us
I20251215 08:59:34.759028 21971 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20251215 08:59:34.764535 23635 consensus_queue.cc:1059] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251215 08:59:34.765416 23825 raft_consensus.cc:3055] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Stepping down as leader of term 8
I20251215 08:59:34.765779 23825 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:34.766314 23635 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:34.767019 23635 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.769975 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 52370.5us
I20251215 08:59:34.780223 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.770260 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:34.783771 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:34.766873 23825 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.755187 23838 log.cc:927] Time spent T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.080s user 0.002s sys 0.000s
I20251215 08:59:34.785035 23825 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:34.786965 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.787742 21973 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:34.788751 21378 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:34.789214 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.789400 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.790103 21971 raft_consensus.cc:2393] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 9: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:34.789247 23796 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:34.790769 23796 raft_consensus.cc:697] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:34.791908 23823 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:34.792193 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:34.792367 23823 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:34.792644 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.791917 23796 consensus_queue.cc:237] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.793373 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.792749 23823 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:34.794193 23801 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.794958 23618 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.795070 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:34.796108 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 114127us
I20251215 08:59:34.797149 21232 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:34.798096 23750 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:34.798905 21232 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:34.793454 21978 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:34.801509 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.802331 21972 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:34.803861 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.804679 21972 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:34.806731 23837 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.807570 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.808440 21972 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:34.809135 23635 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.779598 23784 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.782008 23800 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.807271 23823 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.769775 23813 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:34.810806 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:34.811161 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 82965us
W20251215 08:59:34.807957 23366 consensus_meta.cc:287] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.620s user 0.007s sys 0.011s
I20251215 08:59:34.814529 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.823781 23708 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:34.816473 21307 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:34.835309 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.827239 23750 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:34.838392 23618 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:34.839017 21978 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251215 08:59:34.839246 23618 raft_consensus.cc:697] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:34.846033 23682 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:34.846918 23682 consensus_queue.cc:1243] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer 195a56ac72a84018880bfb3bf6e6ef1d log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251215 08:59:34.847465 23682 raft_consensus.cc:3055] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:34.847764 23682 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:34.840683 23618 consensus_queue.cc:237] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.848771 23682 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.850739 23682 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:34.852809 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:34.854846 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:34.857277 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:34.858129 21335 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:34.860531 23795 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:34.861063 21978 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251215 08:59:34.861297 23795 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.861366 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:34.862650 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.863008 21335 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:34.863550 21978 raft_consensus.cc:2410] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20251215 08:59:34.864413 23795 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.866333 23801 consensus_queue.cc:1059] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:34.867283 23801 raft_consensus.cc:3055] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:34.867619 23801 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:34.873952 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:34.826478 23777 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.892411 23777 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:34.879403 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:34.868486 23801 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.881631 21407 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:34.894016 23801 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.883332 23823 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.887650 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:34.895087 21978 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:34.895771 23796 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:34.896409 23796 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.898438 23796 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:34.890682 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 91965.1us
I20251215 08:59:34.826795 23762 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.913661 23762 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.918605 21978 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:34.920090 21378 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:34.910966 21263 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:34.921379 23826 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:34.931034 23826 raft_consensus.cc:697] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:34.932031 23826 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:34.933593 21262 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:34.937438 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:34.937508 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:34.941375 21265 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:34.941468 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:34.942147 21978 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251215 08:59:34.945322 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:34.945420 21229 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:34.945999 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:34.946805 21258 raft_consensus.cc:2393] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:34.947554 21229 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:34.948506 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:34.947664 21304 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:34.955157 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:34.957993 21262 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251215 08:59:34.955111 23708 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:34.959228 23708 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:34.959594 23708 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:34.961426 23796 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:34.965493 21333 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.965858 23750 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251215 08:59:34.967943 23682 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:34.968647 23682 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:34.969488 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:34.971192 23682 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:34.955334 23366 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20251215 08:59:34.975329 23624 consensus_queue.cc:1059] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:34.976397 23635 raft_consensus.cc:3055] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:34.976809 23635 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:34.977685 23635 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.978962 23635 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:34.980643 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:34.981303 21978 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:34.937984 23762 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:34.986222 23777 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:34.989434 23777 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:34.989916 23777 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:34.991896 23762 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:34.994638 23682 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:34.999296 23618 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.004509 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.012837 21335 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251215 08:59:35.011765 23682 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.015075 21259 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:35.016428 23682 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:35.016993 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:35.016793 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:35.017855 21975 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:35.018111 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.018899 21407 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:35.020900 21162 catalog_manager.cc:5654] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 3, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 3 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:35.022579 21162 catalog_manager.cc:5654] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.024083 21162 catalog_manager.cc:5654] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 6, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 6 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.029968 21265 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:35.008198 21333 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:35.040547 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.040596 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.041587 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:35.042229 21340 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:35.035537 23825 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20251215 08:59:35.042840 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:35.047614 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:35.058328 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.060285 21333 raft_consensus.cc:2410] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:35.061789 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.036147 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:35.064819 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.071197 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:35.075953 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 43505.2us
I20251215 08:59:35.029302 23618 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.105981 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.110466 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:35.112619 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:35.071990 23682 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.123905 21335 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:35.046559 23851 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20251215 08:59:35.128033 23750 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.128940 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.129667 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.130590 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.131507 23708 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.132592 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.134670 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.136381 23796 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:35.137071 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.138450 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.142936 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.090173 23795 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.082837 23847 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.154624 21264 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:35.156702 23846 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:35.158937 23857 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:35.159178 21162 catalog_manager.cc:5654] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.169220 23774 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.090718 21265 raft_consensus.cc:2410] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:35.162948 21304 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:35.194077 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:35.194581 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 103991us
I20251215 08:59:35.148773 21340 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:35.127524 23830 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251215 08:59:35.185120 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 104958us
W20251215 08:59:35.179769 21232 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.894s (SENT) [suppressed 21 similar messages]
W20251215 08:59:35.160969 21163 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36424 (request call id 225) took 20128 ms (client timeout 15000 ms). Trace:
I20251215 08:59:35.203366 21412 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:35.202646 21333 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251215 08:59:35.205757 21336 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:35.202611 21407 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:35.203778 21162 catalog_manager.cc:5654] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.223126 21162 catalog_manager.cc:5654] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.197120 23830 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:35.179342 23682 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.263837 23635 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.264510 23861 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:35.264444 23635 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.200759 23796 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.265386 23861 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.266484 23635 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:35.266317 23796 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.268292 21162 catalog_manager.cc:5654] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 4 to 5, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.225906 23762 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.279788 21162 catalog_manager.cc:5654] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
W20251215 08:59:35.195704 21163 rpcz_store.cc:269] 1215 08:59:15.051299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:15.051610 (+ 311us) service_pool.cc:225] Handling call
1215 08:59:35.151561 (+20099951us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:34.136411 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:34.171535 (+ 35124us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:34.171581 (+ 46us) write_op.cc:435] Acquired schema lock
1215 08:59:34.171604 (+ 23us) tablet.cc:660] Decoding operations
1215 08:59:34.172368 (+ 764us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:34.172457 (+ 89us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:34.172493 (+ 36us) tablet.cc:683] Acquiring locks for 5 operations
1215 08:59:34.173119 (+ 626us) tablet.cc:699] Row locks acquired
1215 08:59:34.173147 (+ 28us) write_op.cc:260] PREPARE: finished
1215 08:59:34.173331 (+ 184us) write_op.cc:270] Start()
1215 08:59:34.173481 (+ 150us) write_op.cc:276] Timestamp: P: 1765789174173262 usec, L: 0
1215 08:59:34.173502 (+ 21us) op_driver.cc:348] REPLICATION: starting
1215 08:59:34.217385 (+ 43883us) log.cc:844] Serialized 2089 byte log entry
1215 08:59:34.644817 (+427432us) op_driver.cc:464] REPLICATION: finished
1215 08:59:34.743359 (+ 98542us) write_op.cc:301] APPLY: starting
1215 08:59:34.743468 (+ 109us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:34.743650 (+ 182us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:34.751164 (+ 7514us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:34.752100 (+ 936us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:34.831509 (+ 79409us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
1215 08:59:34.834118 (+ 2609us) write_op.cc:312] APPLY: finished
1215 08:59:34.855654 (+ 21536us) log.cc:844] Serialized 47 byte log entry
1215 08:59:34.856940 (+ 1286us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:34.857127 (+ 187us) write_op.cc:454] Released schema lock
1215 08:59:34.857181 (+ 54us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:34.861661 (+ 4480us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:34.861691 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1434593,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":81443,"apply.run_cpu_time_us":3538,"apply.run_wall_time_us":118294,"num_ops":5,"prepare.queue_time_us":653931,"prepare.run_cpu_time_us":10509,"prepare.run_wall_time_us":194916,"raft.queue_time_us":269919,"raft.run_cpu_time_us":9164,"raft.run_wall_time_us":119121,"replication_time_us":455721,"spinlock_wait_cycles":1872640,"thread_start_us":1217355,"threads_started":4,"wal-append.queue_time_us":98311}]]}
I20251215 08:59:35.224498 23857 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.294938 23857 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:35.244554 23847 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.296135 23682 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.297088 23847 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.297744 21406 raft_consensus.cc:2410] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20251215 08:59:35.244779 23830 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:35.298607 21412 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:35.205932 23795 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:35.203994 23750 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.306916 23618 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.307616 23750 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.244035 23774 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.234187 23846 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.314082 23846 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.313987 21162 catalog_manager.cc:5654] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 3, leader changed from f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.315034 23862 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:35.315822 23862 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.316047 21162 catalog_manager.cc:5654] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.317705 21162 catalog_manager.cc:5654] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 6, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 6 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.303822 23796 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.305326 23795 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.303071 21413 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:35.321206 21415 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:35.329293 23796 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.332082 23777 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.302045 23861 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.337666 21336 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:35.344559 21230 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:35.353091 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 152395us
I20251215 08:59:35.296938 23682 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:35.358263 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.364717 21162 catalog_manager.cc:5654] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 5, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.308678 23618 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.380473 21264 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:35.385468 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.386307 21264 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:35.391227 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.391315 21162 catalog_manager.cc:5654] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
W20251215 08:59:35.394193 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 49774.5us
I20251215 08:59:35.366444 23682 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:35.375231 23795 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.398934 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:35.406270 21973 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:35.398092 21381 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:35.414070 21381 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:35.415503 21336 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:35.417922 21162 catalog_manager.cc:5654] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.399216 23682 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:35.397864 23777 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.398221 23859 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.395170 23867 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.403792 23837 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:35.392009 23796 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.413662 23745 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.421614 23851 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:35.450448 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:35.447189 21162 catalog_manager.cc:5654] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 6, leader changed from f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 6 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.451085 21978 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:35.414225 23618 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.404189 23795 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.447364 21230 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:35.455227 21333 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:35.455979 21336 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: [4.3-5.4]
I20251215 08:59:35.444407 23708 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.458716 21162 catalog_manager.cc:5654] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.470351 21230 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:35.444600 23867 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.450749 23851 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:35.451104 23774 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.471156 23837 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:35.491801 21162 catalog_manager.cc:5654] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 4 to 7, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 7 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:35.482738 23851 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:35.513494 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:35.515810 21978 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:35.467166 23745 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.520057 21229 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:35.483089 23708 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:35.521276 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.525903 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:35.530398 21333 raft_consensus.cc:1240] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251215 08:59:35.453612 23859 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.494225 23837 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.534832 21413 raft_consensus.cc:2393] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:35.512344 23862 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.557463 23624 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:35.525223 23765 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:35.584875 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 54675.7us
W20251215 08:59:35.586498 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 43848.5us
I20251215 08:59:35.587165 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.530457 23800 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:35.584424 21415 consensus_queue.cc:237] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.580924 21162 catalog_manager.cc:5654] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 4, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 4 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:35.590548 21230 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:35.618155 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:35.544588 23837 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:35.530541 23708 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.634534 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.543277 23774 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.638906 21340 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 4: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
W20251215 08:59:35.640343 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 54123.2us
I20251215 08:59:35.640448 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.586695 23765 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:35.666059 21412 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:35.664271 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.664539 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.667174 21413 raft_consensus.cc:2410] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20251215 08:59:35.667922 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.592233 23366 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:35.670064 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.636873 21162 catalog_manager.cc:5654] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.667047 21407 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:35.668506 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.636284 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:35.672152 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.631994 23862 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:35.661329 23708 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:35.671826 21976 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:35.672371 21408 raft_consensus.cc:2376] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4. Re-sending same reply.
I20251215 08:59:35.675877 21410 raft_consensus.cc:2393] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:35.682788 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.682739 21162 catalog_manager.cc:5654] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 4, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 4 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } }
I20251215 08:59:35.635018 23867 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.685024 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.683701 21333 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:35.687968 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.688694 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:35.688185 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 50554.3us
I20251215 08:59:35.689898 21406 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:35.667783 23765 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:35.695220 21336 raft_consensus.cc:1275] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251215 08:59:35.698225 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.699086 21259 raft_consensus.cc:1240] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:35.699560 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.695679 21336 pending_rounds.cc:85] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 1
I20251215 08:59:35.631448 23624 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.691251 23366 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.700959 21336 pending_rounds.cc:107] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251215 08:59:35.706629 21336 raft_consensus.cc:2889] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:35.705159 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:35.688689 23796 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.710579 21406 raft_consensus.cc:2410] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20251215 08:59:35.713708 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.709830 23745 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.697527 23795 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.714900 23587 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:35.716281 23871 log.cc:927] Time spent T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20251215 08:59:35.734431 21410 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:35.738719 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 148245us
I20251215 08:59:35.699430 23618 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:35.723471 21162 catalog_manager.cc:5654] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 4, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 4 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:35.746311 21230 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:35.701704 23867 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:35.686111 23862 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.691674 23774 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:35.762986 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:35.763612 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 50025.4us
W20251215 08:59:35.753245 23870 log.cc:927] Time spent T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20251215 08:59:35.765664 21334 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251215 08:59:35.767081 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.767625 21230 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:35.766448 21334 pending_rounds.cc:85] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:35.769476 21334 pending_rounds.cc:107] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251215 08:59:35.768778 21162 catalog_manager.cc:5654] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.769724 21334 raft_consensus.cc:2889] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:35.772603 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.773089 21406 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:35.773918 23867 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.774403 23867 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.774861 23774 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.776374 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.776762 23867 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.777792 21230 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:35.777488 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.773428 21162 catalog_manager.cc:5654] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:35.761768 23795 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:35.782768 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:35.776932 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.784991 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.785923 21259 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251215 08:59:35.715363 23872 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:35.766330 23800 raft_consensus.cc:697] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:35.799984 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.800318 23800 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.801254 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:35.802420 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:35.802281 23851 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.803352 21408 raft_consensus.cc:2368] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:35.709192 23873 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.789057 23848 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.790722 21379 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:35.793150 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:35.790925 21336 raft_consensus.cc:1240] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:35.674567 23864 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:35.789139 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.799242 23862 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.771485 23587 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:35.788568 21341 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:35.810318 21334 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:35.804090 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:35.815173 21410 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20251215 08:59:35.819332 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:35.819675 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:35.820516 21978 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:35.809161 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.781556 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:35.830174 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:35.835296 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.781754 23014 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20251215 08:59:35.777663 23879 consensus_queue.cc:1059] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20251215 08:59:35.870354 21307 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.823673 23618 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:35.785156 23863 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20251215 08:59:35.873425 21976 raft_consensus.cc:1275] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20251215 08:59:35.865988 21232 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.014s sys 0.025s
I20251215 08:59:35.870616 21162 catalog_manager.cc:5654] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } }
I20251215 08:59:35.814215 23867 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:35.877156 21946 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:35.829663 23862 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:35.864190 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:35.808501 23859 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:35.812530 23774 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:35.883880 21232 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.678s user 0.014s sys 0.025s
W20251215 08:59:35.895671 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 723693us
I20251215 08:59:35.837122 23762 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:35.896776 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 106117us
I20251215 08:59:35.834635 23587 raft_consensus.cc:697] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:35.893994 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:35.897807 21379 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.899325 21259 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:35.899395 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 60435us
I20251215 08:59:35.871661 23851 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:35.827899 23848 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.864084 23889 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:35.917594 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.839174 23750 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.061s
I20251215 08:59:35.924808 21946 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.886585 21975 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
W20251215 08:59:35.961885 21973 consensus_meta.cc:287] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.542s user 0.008s sys 0.002s
I20251215 08:59:35.963481 21973 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:35.897409 23796 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:35.892858 23366 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:35.963995 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 93724.4us
I20251215 08:59:35.809161 23830 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:35.898845 23869 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:35.969398 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.970113 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.971911 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.888136 23857 consensus_queue.cc:1059] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20251215 08:59:35.918176 23813 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:35.827495 23872 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.990648 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.985734 21413 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:35.994818 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.838954 23873 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.907052 23762 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.010865 21263 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:36.010946 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:36.013312 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:35.877426 23864 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:35.986820 23745 raft_consensus.cc:2764] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20251215 08:59:36.015811 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.998230 23624 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:36.018385 21258 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:35.996702 23366 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:36.032737 21260 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:36.029338 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:35.998257 23868 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:35.994463 23813 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:36.034603 21337 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:35.993958 23880 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:36.037956 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 137642us
I20251215 08:59:35.992806 23765 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.034047 21257 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
W20251215 08:59:36.055855 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 52288.8us
I20251215 08:59:36.041612 23889 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:36.035444 23366 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:36.064587 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:36.046866 23813 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.065605 23892 consensus_queue.cc:1048] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
W20251215 08:59:36.069232 23878 log.cc:927] Time spent T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.284s user 0.000s sys 0.001s
W20251215 08:59:36.072259 23877 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.296s user 0.000s sys 0.001s
I20251215 08:59:36.033922 23885 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:36.045873 23880 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:36.051936 23874 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.077950 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:36.053915 23868 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.074330 23366 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:36.090212 23864 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.092095 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:36.095763 23708 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.096933 23848 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:36.097380 21334 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:36.097587 23873 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.098248 23762 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.098925 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.099485 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.083943 23892 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20251215 08:59:36.100455 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.101197 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.103346 21259 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:36.106681 23868 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.084297 23874 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.108100 23830 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.079392 23880 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.180920 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.182240 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.185772 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.186375 21335 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:36.186654 23874 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.173333 23765 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:36.036015 23587 consensus_queue.cc:237] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.081794 23885 raft_consensus.cc:697] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:36.192013 23885 consensus_queue.cc:237] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.194365 23708 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.172617 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.114375 23837 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:36.117177 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 207206us
I20251215 08:59:36.117933 21379 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:36.175802 21230 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:36.108150 23813 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.176335 21333 raft_consensus.cc:2376] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4. Re-sending same reply.
I20251215 08:59:36.177199 21338 raft_consensus.cc:2393] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:36.180107 23774 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:36.196197 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.200898 21378 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:36.222777 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 50268.9us
W20251215 08:59:36.224944 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 49227.3us
I20251215 08:59:36.231364 21334 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:36.232023 23900 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:36.232339 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:36.234057 23813 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.234468 21232 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:36.234647 23813 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:36.145462 23847 consensus_queue.cc:1059] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20251215 08:59:36.238322 23901 consensus_queue.cc:1059] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20251215 08:59:36.240665 21340 raft_consensus.cc:1275] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251215 08:59:36.241053 21340 pending_rounds.cc:85] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:36.241291 21340 pending_rounds.cc:107] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251215 08:59:36.241518 21340 raft_consensus.cc:2889] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:36.242187 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.243731 23745 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.244207 23745 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.246276 23745 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.117812 23864 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:36.247146 23864 raft_consensus.cc:697] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:36.248049 23864 consensus_queue.cc:237] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.113555 23889 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:36.254186 21265 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:36.255883 21336 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:36.255219 23837 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.259123 21974 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:36.264503 21334 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
W20251215 08:59:36.283569 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 165729us
I20251215 08:59:36.284621 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:36.260953 23682 consensus_meta.cc:287] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.820s user 0.007s sys 0.003s
I20251215 08:59:36.226912 23708 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:36.317787 23682 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.318508 23700 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:36.279338 23868 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:36.333804 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 50407.1us
W20251215 08:59:36.319794 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 118982us
W20251215 08:59:36.330442 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 45953us
I20251215 08:59:36.335477 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:36.321297 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:36.279680 23762 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:36.336068 21259 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:36.337222 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 102823us
I20251215 08:59:36.339282 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:36.318753 23847 raft_consensus.cc:3055] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:36.320451 23765 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:36.343969 23762 raft_consensus.cc:697] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:36.336378 23868 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:36.320624 23879 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.348068 21307 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.283196 23873 consensus_queue.cc:1059] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20251215 08:59:36.328346 23872 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:36.377691 23883 log.cc:927] Time spent T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.603s user 0.002s sys 0.000s
I20251215 08:59:36.347450 23847 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:36.379351 23859 raft_consensus.cc:3055] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:36.379757 23859 raft_consensus.cc:740] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:36.380625 23869 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:36.359398 23868 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:36.388296 23882 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.613s user 0.004s sys 0.000s
I20251215 08:59:36.320079 23682 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:36.321527 23903 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:36.354099 23897 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20251215 08:59:36.355618 23902 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:36.353467 21229 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.339917 23885 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:36.362561 23762 consensus_queue.cc:237] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.365402 23901 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.346789 23796 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:36.370685 21259 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
W20251215 08:59:36.375631 23884 log.cc:927] Time spent T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.601s user 0.002s sys 0.000s
I20251215 08:59:36.378912 23847 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:36.432394 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 84369.4us
I20251215 08:59:36.433600 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:36.346659 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 120189us
I20251215 08:59:36.393659 21974 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:36.394198 21232 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:36.395176 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.397625 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.398542 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.398831 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.397688 23902 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:36.397456 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 44108.8us
I20251215 08:59:36.377737 23873 raft_consensus.cc:3055] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:36.453409 23873 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:36.454305 23873 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.455566 23873 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.455750 23902 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:36.394308 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 51811.7us
I20251215 08:59:36.410293 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1911) took 1051 ms. Trace:
I20251215 08:59:36.434635 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:36.394840 23903 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:36.458988 21307 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.396991 23867 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:36.457593 21262 rpcz_store.cc:276] 1215 08:59:35.358826 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:35.358977 (+ 151us) service_pool.cc:225] Handling call
1215 08:59:35.384467 (+ 25490us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:35.528112 (+143645us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:35.528145 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:35.558418 (+ 30273us) log.cc:844] Serialized 26 byte log entry
1215 08:59:35.687496 (+129078us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:35.687524 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:35.687581 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:36.387625 (+700044us) spinlock_profiling.cc:243] Waited 9.57 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:36.399060 (+ 11435us) raft_consensus.cc:1685] finished
1215 08:59:36.399082 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:36.410217 (+ 11135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17772,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":15905,"spinlock_wait_cycles":22004480,"thread_start_us":182976,"threads_started":2,"wal-append.queue_time_us":175579}
I20251215 08:59:36.461617 23889 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:36.461797 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:36.462882 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 101235us
W20251215 08:59:36.463749 21944 leader_election.cc:336] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.837s (SENT)
I20251215 08:59:36.400694 23874 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:36.467516 21944 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:36.476193 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.477351 21336 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:36.482199 21307 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:36.483759 21230 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:36.433126 23893 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.491812 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:36.491991 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.492609 21406 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:36.493605 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.493889 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.494684 21412 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:36.496325 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:36.457530 23682 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:36.393573 23879 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.506366 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:36.503149 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.507386 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.515130 21332 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:36.515560 21341 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.509047 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.516479 21336 raft_consensus.cc:2393] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:36.509913 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.380754 23859 consensus_queue.cc:260] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.521943 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:36.460741 23903 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.465871 23774 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:36.504657 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.527441 21407 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:36.502789 23813 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:36.520720 23750 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:36.521304 23874 raft_consensus.cc:697] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:36.522496 21412 raft_consensus.cc:2376] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4. Re-sending same reply.
I20251215 08:59:36.523161 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.523134 21232 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:36.466799 23889 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:36.523468 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:36.525686 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:36.513651 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 100501us
W20251215 08:59:36.527534 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 63823.3us
I20251215 08:59:36.520784 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:36.541460 21409 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.542096 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:36.542678 21974 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:36.528934 21336 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:36.480685 23800 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:36.543954 21304 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.466001 23825 consensus_queue.cc:1048] T 5116dbb730274ac9982610c7f5f8f5cd P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20251215 08:59:36.553231 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.522370 23830 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.559602 23830 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.562434 23830 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.460150 23901 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.582510 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:36.567266 23874 consensus_queue.cc:237] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.526299 23796 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.567518 21977 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:36.572743 21974 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:36.551344 23889 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:36.549151 23847 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:36.586340 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.586602 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.585417 21406 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:36.552062 23846 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:36.529893 23750 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:36.603229 21944 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:36.607689 21409 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:36.581866 23795 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:36.565378 23708 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:36.610736 21974 raft_consensus.cc:1275] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:36.620009 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.623528 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.511477 23893 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.637825 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.638474 21413 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
W20251215 08:59:36.642123 23909 log.cc:927] Time spent T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20251215 08:59:36.644613 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.610639 23846 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:36.646365 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 94480.1us
I20251215 08:59:36.621873 23750 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.653090 21338 raft_consensus.cc:2393] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:36.653607 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.624747 23859 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.655166 21946 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.624293 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:36.627779 23800 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:36.591342 23813 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20251215 08:59:36.665220 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:36.666862 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 109481us
I20251215 08:59:36.667546 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.671037 21413 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:36.663594 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.657647 23846 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:36.676514 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.630074 23796 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.678335 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.678228 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.691208 21335 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:36.658447 23795 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:36.686095 23366 consensus_meta.cc:287] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.514s user 0.009s sys 0.000s
I20251215 08:59:36.688175 23900 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:36.696472 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.700138 21341 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:36.702831 21379 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:36.704703 21335 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:36.690141 23813 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:36.710479 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.712018 21332 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:36.706640 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:36.636999 23880 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.713425 21406 raft_consensus.cc:2368] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:36.714874 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.721954 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.722782 21413 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:36.700978 23900 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:36.702915 23800 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:36.708009 23867 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:36.710171 23848 raft_consensus.cc:2804] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:36.712692 23902 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:36.716857 23708 raft_consensus.cc:2764] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20251215 08:59:36.739274 23900 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.731324 23813 raft_consensus.cc:697] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:36.751657 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:36.755096 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 234585us
I20251215 08:59:36.701905 23366 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.747176 21946 leader_election.cc:400] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:36.806721 21946 leader_election.cc:403] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20251215 08:59:36.807415 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 90505.5us
I20251215 08:59:36.807897 23366 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:36.808552 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.808704 23830 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:36.809629 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:36.809950 21408 raft_consensus.cc:2393] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:36.745797 23880 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:36.811415 23880 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:36.811868 23880 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.812275 21413 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:36.813110 23708 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:36.813690 23708 raft_consensus.cc:697] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:36.813795 23813 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.813704 23896 raft_consensus.cc:3055] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:36.731551 23837 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20251215 08:59:36.815814 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:36.817554 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:36.769430 23919 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:36.819446 23919 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.819876 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.820927 23880 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.773376 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.759187 23848 raft_consensus.cc:697] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:36.781163 23795 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.797744 23893 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:36.798476 23901 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:36.759342 23879 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:36.814909 23896 raft_consensus.cc:740] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:36.764791 23907 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:36.817974 23745 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:36.821805 23917 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:36.822335 23868 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.822683 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:36.823576 21338 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:36.778029 23905 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.277s user 0.001s sys 0.000s
I20251215 08:59:36.817101 21378 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:36.770953 21406 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:36.815369 23902 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.853914 21230 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:36.858559 23846 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.755637 23867 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.832800 23864 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:36.890563 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.896452 21379 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:59:36.901136 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 84113.8us
I20251215 08:59:36.832494 23892 consensus_queue.cc:1048] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20251215 08:59:36.884059 23745 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:36.910029 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:36.910161 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 62811.5us
I20251215 08:59:36.913233 23887 raft_consensus.cc:2804] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:36.913671 23887 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:36.914165 23887 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:36.915043 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:36.913611 23918 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:36.890008 23917 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:36.892854 23762 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:36.892661 23907 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:36.915634 21406 raft_consensus.cc:606] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:36.904986 23800 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:36.940847 23907 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.941489 23800 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:36.942032 23800 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:36.943025 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 109253us
I20251215 08:59:36.911120 23863 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:36.944238 23863 raft_consensus.cc:697] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:36.944144 23911 consensus_queue.cc:1059] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:36.916180 23859 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20251215 08:59:36.945211 23896 consensus_queue.cc:260] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.946198 23896 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:36.946668 23911 raft_consensus.cc:3055] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:36.947145 23911 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:36.911386 23708 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.950629 23902 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:36.952188 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:36.934288 23796 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:36.935900 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:36.915972 23745 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.953583 23848 consensus_queue.cc:237] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.955801 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.956454 21336 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:36.925431 23765 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:36.906914 23864 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:36.957449 23864 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:36.960615 23867 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:36.951830 23880 consensus_queue.cc:1175] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" responder_term: 8 status { last_received { term: 7 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251215 08:59:36.963003 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:36.931473 23846 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:36.939394 23919 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:36.962092 23868 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:36.966269 23918 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:36.966733 23918 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:36.967566 23863 consensus_queue.cc:237] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:36.971740 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:36.972297 21974 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.972959 23869 consensus_queue.cc:1048] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:36.973338 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.974429 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.977205 21415 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
W20251215 08:59:36.977909 23873 consensus_meta.cc:287] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.522s user 0.011s sys 0.000s
I20251215 08:59:36.978479 23919 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:36.979018 23919 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:36.980449 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 55815.6us
I20251215 08:59:36.962337 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:36.960389 23889 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:36.992128 21946 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
W20251215 08:59:36.992574 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 282335us
I20251215 08:59:36.999735 21974 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:37.000669 21335 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251215 08:59:37.001497 21232 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:36.990311 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:37.005931 21265 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:37.007007 21259 raft_consensus.cc:1240] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:36.982412 23762 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:37.008062 21336 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:37.009554 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:37.010815 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:36.948161 23911 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.014144 23911 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:37.016191 23796 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:37.016808 23796 raft_consensus.cc:697] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:37.017956 23796 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.027580 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:36.983031 23917 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:37.010394 23873 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20251215 08:59:36.970957 23750 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.040305 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:37.041181 21415 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:37.042658 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:59:37.047932 23933 log.cc:927] Time spent T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20251215 08:59:37.064337 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:37.038138 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:37.065738 21408 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:36.977125 23868 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:37.057787 21332 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:37.058175 21337 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:37.060185 21379 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:37.061535 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:37.069397 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:37.069993 21265 raft_consensus.cc:3060] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:37.008668 23635 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:37.033311 23893 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:37.118585 21258 raft_consensus.cc:1240] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:37.125643 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:37.126205 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:37.012703 23867 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20251215 08:59:37.449764 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:37.610846 23930 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.006s
I20251215 08:59:37.173446 23925 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:37.617285 21257 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:37.129882 23635 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.618625 21406 raft_consensus.cc:1275] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251215 08:59:37.127117 23903 consensus_meta.cc:287] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.570s user 0.010s sys 0.001s
I20251215 08:59:37.137398 21265 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:37.472147 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:37.620604 23903 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.609290 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:37.175362 23846 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:37.626775 23846 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.627727 23903 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:37.175343 23868 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.628753 23846 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:37.615310 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 613870us
I20251215 08:59:37.632688 23868 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:37.635516 23762 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:37.534662 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:37.451148 23864 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.639544 23864 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:37.545445 23917 consensus_meta.cc:287] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.516s user 0.010s sys 0.000s
I20251215 08:59:37.640477 23917 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.643388 23917 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:37.541466 23934 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.493s user 0.000s sys 0.002s
I20251215 08:59:37.535557 23750 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:37.262524 23928 log.cc:927] Time spent T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.300s user 0.001s sys 0.000s
W20251215 08:59:37.551755 23896 consensus_meta.cc:287] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.605s user 0.011s sys 0.000s
W20251215 08:59:37.602671 23800 consensus_meta.cc:287] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.660s user 0.009s sys 0.002s
I20251215 08:59:37.173100 23923 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:37.448179 23936 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.243s user 0.000s sys 0.000s
I20251215 08:59:37.071219 23869 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:37.206115 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 146048us
I20251215 08:59:37.625186 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:37.425700 23745 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:37.438678 23795 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.008s sys 0.004s
I20251215 08:59:37.617311 23925 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:37.628317 23635 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:37.610960 21230 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:37.719214 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 108337us
W20251215 08:59:37.643663 23941 log.cc:927] Time spent T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.219s user 0.003s sys 0.007s
W20251215 08:59:37.543211 23887 consensus_meta.cc:287] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.629s user 0.008s sys 0.000s
I20251215 08:59:37.070953 23873 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:37.732878 23873 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:37.732836 23887 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.644744 23893 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.437400 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:37.734879 23887 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:37.735184 21259 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 1.
W20251215 08:59:37.475950 23931 log.cc:927] Time spent T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.511s user 0.002s sys 0.000s
W20251215 08:59:37.483373 23918 consensus_meta.cc:287] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.516s user 0.009s sys 0.000s
I20251215 08:59:37.736739 23918 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.738746 23918 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:37.547313 21408 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:37.246997 23851 raft_consensus.cc:2804] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:37.739877 23851 raft_consensus.cc:697] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:37.740823 23851 consensus_queue.cc:237] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.694334 23894 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:37.681208 21944 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:37.683374 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 557203us
I20251215 08:59:37.685447 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:37.687306 21266 raft_consensus.cc:3060] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
W20251215 08:59:37.666625 23900 consensus_meta.cc:287] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.888s user 0.012s sys 0.005s
I20251215 08:59:37.648223 23874 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:37.687466 23903 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:37.711331 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:37.762853 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:37.763129 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 51940.9us
I20251215 08:59:37.763623 21412 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:37.764894 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:37.764824 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:37.766448 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 85433.7us
I20251215 08:59:37.718277 23846 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:37.720132 21230 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:37.656342 23889 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:37.723218 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:37.725018 21411 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:37.726689 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1758) took 1253 ms. Trace:
I20251215 08:59:37.687815 23800 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.741199 23917 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:37.741859 21338 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:37.115559 23587 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:37.813647 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 93622.5us
I20251215 08:59:37.206773 23919 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:37.126843 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:37.854962 21415 raft_consensus.cc:606] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:37.695350 23923 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.855472 21415 raft_consensus.cc:614] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:37.760886 21408 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:37.774834 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:37.720286 23925 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:37.720100 21340 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.006s sys 0.009s
W20251215 08:59:37.857316 21340 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.965s user 0.006s sys 0.009s
I20251215 08:59:37.778033 23762 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:37.778949 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:37.780850 21334 rpcz_store.cc:276] 1215 08:59:36.473546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:36.473696 (+ 150us) service_pool.cc:225] Handling call
1215 08:59:36.474127 (+ 431us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:36.474568 (+ 441us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:36.474608 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:36.475177 (+ 569us) log.cc:844] Serialized 26 byte log entry
1215 08:59:36.847864 (+372687us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b1800264fa0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:37.077451 (+229587us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:37.224888 (+147437us) log.cc:844] Serialized 14 byte log entry
1215 08:59:37.225172 (+ 284us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:37.225284 (+ 112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:37.264584 (+ 39300us) spinlock_profiling.cc:243] Waited 7.53 ms on lock 0x7b1800264fa0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:37.726472 (+461888us) raft_consensus.cc:1685] finished
1215 08:59:37.726495 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:37.726616 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5555,"prepare.queue_time_us":167909,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":7952,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":34188,"spinlock_wait_cycles":20112384,"thread_start_us":229408,"threads_started":1,"wal-append.queue_time_us":1211732}
I20251215 08:59:37.733428 23873 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:37.786355 21381 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:37.749990 23896 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:37.814869 21230 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:37.870287 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 55496.4us
I20251215 08:59:37.872224 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 252) took 1165 ms. Trace:
I20251215 08:59:37.734882 23893 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:37.780380 23830 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:37.867290 21975 raft_consensus.cc:2393] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:37.806020 23918 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:37.872565 21410 rpcz_store.cc:276] 1215 08:59:36.706547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:36.707038 (+ 491us) service_pool.cc:225] Handling call
1215 08:59:36.707423 (+ 385us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:36.707877 (+ 454us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:36.707917 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:36.718001 (+ 10084us) log.cc:844] Serialized 26 byte log entry
1215 08:59:36.845296 (+127295us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:36.845334 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:36.845393 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:37.261922 (+416529us) spinlock_profiling.cc:243] Waited 94.2 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:37.859214 (+597292us) raft_consensus.cc:1685] finished
1215 08:59:37.859235 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:37.872164 (+ 12929us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":11883,"spinlock_wait_cycles":216650624,"thread_start_us":103528,"threads_started":1,"wal-append.queue_time_us":221101}
I20251215 08:59:37.777592 23795 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:37.717805 23869 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:37.877259 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 91047.1us
I20251215 08:59:37.857717 23930 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:37.878103 21262 raft_consensus.cc:2368] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7.
I20251215 08:59:37.881361 23830 raft_consensus.cc:697] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:37.907058 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:37.907938 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:37.769042 23900 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.875259 23587 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:37.765419 23894 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.766935 23874 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:37.813313 23846 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.858613 23917 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:37.897713 23897 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:37.978559 23930 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:37.896166 23864 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:38.011574 21976 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:37.985219 23868 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:37.881690 23762 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:37.965184 23934 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251215 08:59:37.858215 23915 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:38.004679 23945 log.cc:927] Time spent T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251215 08:59:38.005232 21341 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:38.006544 23917 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.002229 23897 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:37.997179 23918 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
W20251215 08:59:37.971585 23708 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.026s sys 0.029s
W20251215 08:59:37.974506 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 88696.9us
I20251215 08:59:38.022629 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:38.012511 23863 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.042s user 0.012s sys 0.013s
I20251215 08:59:37.981348 23903 consensus_queue.cc:1059] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20251215 08:59:38.009415 23932 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.006s sys 0.015s
W20251215 08:59:38.025655 21946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.000s sys 0.034s
W20251215 08:59:38.090286 21946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.000s sys 0.034s
I20251215 08:59:38.011673 23930 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:38.002429 23919 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:38.052919 21341 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:38.056500 21334 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:38.118039 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:38.071283 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:38.071377 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:38.124781 21259 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:38.119531 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:38.126421 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:38.057332 21943 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.005s sys 0.032s
I20251215 08:59:38.098599 23869 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:38.057363 23708 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.075s user 0.026s sys 0.030s
I20251215 08:59:38.054025 23897 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:38.057564 23863 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.087s user 0.012s sys 0.013s
I20251215 08:59:38.118549 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:38.185746 21411 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:38.093468 23917 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:38.185781 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.025401 23868 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:38.188639 21973 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251215 08:59:38.094491 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:38.015898 23879 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:38.127740 21415 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:38.174306 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.144831 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:38.176064 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:38.177327 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.058655 23745 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.181242 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.020550 23864 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:38.204371 23934 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
W20251215 08:59:38.090173 23932 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.915s user 0.006s sys 0.015s
I20251215 08:59:38.126701 21971 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:38.053964 23889 consensus_queue.cc:1048] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
W20251215 08:59:38.100315 23911 consensus_meta.cc:287] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 1.080s user 0.008s sys 0.012s
I20251215 08:59:38.043668 23915 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:38.232964 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 376020us
I20251215 08:59:38.183784 23830 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:38.094566 23800 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.184927 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:38.266060 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.266800 21263 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:38.267793 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.183630 21260 raft_consensus.cc:1240] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: []
W20251215 08:59:38.145594 23948 log.cc:927] Time spent T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.184s user 0.000s sys 0.001s
W20251215 08:59:38.184083 21943 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.007s sys 0.035s
I20251215 08:59:38.242079 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:38.242204 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.184378 23923 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.301415 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:38.174196 23867 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.256168 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.061079 23918 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:38.323382 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.100163 23848 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:38.324080 23848 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:38.326262 23848 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:38.326424 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1501) took 1289 ms. Trace:
I20251215 08:59:38.326833 23848 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:38.260375 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1756) took 1788 ms. Trace:
I20251215 08:59:38.186496 23869 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:38.239611 23879 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:38.287297 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:38.297369 21266 consensus_meta.cc:287] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.544s user 0.008s sys 0.002s
I20251215 08:59:38.289059 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:38.299660 23946 log.cc:927] Time spent T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.334s user 0.001s sys 0.000s
I20251215 08:59:38.153172 23919 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:38.245389 23953 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.237090 23955 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.319610 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:38.324352 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:38.298576 23868 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.233595 23917 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:38.794591 21380 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.011s sys 0.031s
W20251215 08:59:38.369778 21379 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.052s sys 0.113s
W20251215 08:59:38.928639 21380 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s user 0.016s sys 0.039s
I20251215 08:59:38.913964 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:38.916812 23889 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:38.531356 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:38.929692 23984 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.930064 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.930843 21261 raft_consensus.cc:2368] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
W20251215 08:59:38.931492 21262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 1940) took 1877 ms (client timeout 1523 ms). Trace:
W20251215 08:59:38.931887 21262 rpcz_store.cc:269] 1215 08:59:37.053784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.067764 (+ 13980us) service_pool.cc:225] Handling call
1215 08:59:38.931297 (+1863533us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54000f7654. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:38.931408 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":596263680}
I20251215 08:59:38.932425 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.933053 23917 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:38.933463 21261 raft_consensus.cc:2410] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:38.933605 23917 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:38.934039 21974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 591) took 1732 ms. Trace:
I20251215 08:59:38.934389 21974 rpcz_store.cc:276] 1215 08:59:37.201886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.202484 (+ 598us) service_pool.cc:225] Handling call
1215 08:59:37.682696 (+480212us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:37.884032 (+201336us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:37.884072 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:37.884127 (+ 55us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:37.884168 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:38.933788 (+1049620us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54003f11d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:38.933833 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:38.933963 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":462412160}
I20251215 08:59:38.935037 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:38.936002 21261 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:38.934165 23917 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:38.938699 23917 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.938831 21262 raft_consensus.cc:1275] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:38.939330 23917 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.939944 23917 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:38.519522 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 461093us
W20251215 08:59:38.911984 21334 consensus_meta.cc:287] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.789s user 0.048s sys 0.056s
I20251215 08:59:38.941570 21334 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:38.943097 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:38.944898 21266 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:38.945753 23846 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:38.948683 23846 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:38.309738 23950 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:38.948951 23958 consensus_queue.cc:1059] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:38.929698 23889 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:38.952057 23919 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:38.953320 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:38.326746 21413 rpcz_store.cc:276] 1215 08:59:37.036810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.037234 (+ 424us) service_pool.cc:225] Handling call
1215 08:59:37.070715 (+ 33481us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:37.177428 (+106713us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:37.177476 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:37.634928 (+457452us) log.cc:844] Serialized 26 byte log entry
1215 08:59:38.015692 (+380764us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:38.015749 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:38.016666 (+ 917us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:38.326199 (+309533us) raft_consensus.cc:1685] finished
1215 08:59:38.326218 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:38.326355 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77839,"raft.queue_time_us":78251,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":24447,"thread_start_us":380402,"threads_started":1,"wal-append.queue_time_us":626505}
I20251215 08:59:38.334287 23874 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:38.915416 23925 consensus_meta.cc:287] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 1.040s user 0.010s sys 0.001s
I20251215 08:59:38.828997 23915 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:38.914324 23873 consensus_meta.cc:287] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 1.036s user 0.011s sys 0.000s
I20251215 08:59:38.954535 23869 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:38.803517 23745 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:38.336306 23923 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:38.955442 21406 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.018s sys 0.033s
I20251215 08:59:38.955811 21259 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:38.881844 23907 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.335400 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:38.520040 21337 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.000s sys 0.019s
I20251215 08:59:38.335063 23894 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:38.928985 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.958758 21980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 590) took 1976 ms. Trace:
W20251215 08:59:38.959292 23708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.003s sys 0.006s
I20251215 08:59:38.339641 23867 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:38.743194 23949 log.cc:927] Time spent T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.346s user 0.000s sys 0.000s
I20251215 08:59:38.748502 23960 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:38.297399 23944 log.cc:927] Time spent T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.334s user 0.001s sys 0.001s
W20251215 08:59:38.961187 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:38.963413 23976 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:38.744017 23964 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.327368 23848 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:38.961418 21263 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:38.969699 23813 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:38.713517 21407 consensus_meta.cc:287] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.590s user 0.010s sys 0.000s
I20251215 08:59:39.085295 21407 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:38.706243 21971 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:39.085673 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:38.960035 21340 rpcz_store.cc:276] 1215 08:59:36.472296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:36.473059 (+ 763us) service_pool.cc:225] Handling call
1215 08:59:36.473419 (+ 360us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:36.510399 (+ 36980us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:36.510443 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:36.519415 (+ 8972us) log.cc:844] Serialized 52 byte log entry
1215 08:59:36.886487 (+367072us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:37.490128 (+603641us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:37.857738 (+367610us) log.cc:844] Serialized 14 byte log entry
1215 08:59:37.882192 (+ 24454us) log.cc:844] Serialized 14 byte log entry
1215 08:59:37.958162 (+ 75970us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:38.260073 (+301911us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b54001243d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:38.260130 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:38.260170 (+ 40us) raft_consensus.cc:1685] finished
1215 08:59:38.260181 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:38.260314 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112966,"prepare.queue_time_us":206341,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":216071,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":19919,"spinlock_wait_cycles":73438720,"thread_start_us":1270292,"threads_started":2,"wal-append.queue_time_us":485766}
W20251215 08:59:38.975030 21946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.000s sys 0.055s
W20251215 08:59:39.086869 21946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.894s user 0.000s sys 0.055s
W20251215 08:59:38.519907 21258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.014s sys 0.005s
I20251215 08:59:38.485798 21411 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:38.976619 23917 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.976615 21414 raft_consensus.cc:2393] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:38.611405 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1952) took 1014 ms. Trace:
W20251215 08:59:38.632431 23903 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.002s sys 0.009s
I20251215 08:59:38.341343 23900 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:38.982595 23955 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:38.930315 23984 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:38.988250 23981 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:38.989385 23868 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:38.991227 21976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46170 (request call id 471) took 2055 ms (client timeout 2411 ms). Trace:
I20251215 08:59:39.008848 23979 consensus_queue.cc:1059] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:39.014694 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:39.117060 21976 rpcz_store.cc:269] 1215 08:59:36.935493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:36.935627 (+ 134us) service_pool.cc:225] Handling call
1215 08:59:38.991050 (+2055423us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54004204d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:38.991158 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2422782720}
W20251215 08:59:38.928839 21379 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.155s user 0.070s sys 0.158s
W20251215 08:59:39.119062 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 1.345e+06us
W20251215 08:59:39.082288 21406 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s user 0.025s sys 0.035s
W20251215 08:59:39.082823 21337 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.087s user 0.000s sys 0.030s
I20251215 08:59:39.085769 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1507) took 1015 ms. Trace:
W20251215 08:59:39.087859 21258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.095s user 0.014s sys 0.008s
I20251215 08:59:39.088032 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1506) took 1017 ms. Trace:
I20251215 08:59:39.088595 23950 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:39.087355 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 894721us
I20251215 08:59:39.086629 21263 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:39.095753 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1509) took 1024 ms. Trace:
I20251215 08:59:39.127030 21946 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:39.086608 23953 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.099383 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:39.128733 21413 raft_consensus.cc:606] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:39.093122 23976 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.107528 23873 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.136533 23873 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.137477 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:39.109666 21975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 644) took 2292 ms (client timeout 2665 ms). Trace:
I20251215 08:59:39.138072 23873 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:39.138299 21975 rpcz_store.cc:269] 1215 08:59:36.817714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:36.822402 (+ 4688us) service_pool.cc:225] Handling call
1215 08:59:39.109438 (+2287036us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54005fdf54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:39.109582 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1992540928}
I20251215 08:59:39.138588 23873 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.139899 23984 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.140703 23873 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:39.085620 23708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.779s user 0.003s sys 0.009s
I20251215 08:59:39.115428 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:39.153379 21259 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:39.050391 23983 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.016070 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.121050 21407 rpcz_store.cc:276] 1215 08:59:38.070651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.070805 (+ 154us) service_pool.cc:225] Handling call
1215 08:59:39.085710 (+1014905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:39.121279 21411 rpcz_store.cc:276] 1215 08:59:38.070182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.070325 (+ 143us) service_pool.cc:225] Handling call
1215 08:59:39.087975 (+1017650us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:39.111941 23903 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.012s user 0.002s sys 0.010s
I20251215 08:59:39.084410 21980 rpcz_store.cc:276] 1215 08:59:36.982482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:36.983434 (+ 952us) service_pool.cc:225] Handling call
1215 08:59:37.026857 (+ 43423us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:37.027354 (+ 497us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:37.027419 (+ 65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:37.028658 (+ 1239us) log.cc:844] Serialized 501 byte log entry
1215 08:59:37.305719 (+277061us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:37.305780 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:37.305885 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:38.301914 (+996029us) raft_consensus.cc:1685] finished
1215 08:59:38.301935 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:38.958701 (+656766us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":276528,"threads_started":1,"wal-append.queue_time_us":788506}
W20251215 08:59:39.173233 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 228757us
I20251215 08:59:39.091367 23915 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.126641 21414 rpcz_store.cc:276] 1215 08:59:38.071687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.223552 (+151865us) service_pool.cc:225] Handling call
1215 08:59:39.095669 (+872117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:39.102252 23897 consensus_meta.cc:287] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.862s user 0.007s sys 0.003s
I20251215 08:59:39.118438 23981 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.130641 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:39.130759 23966 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:39.128268 23868 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.152760 23919 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:39.131738 23986 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.132158 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1954) took 1437 ms. Trace:
I20251215 08:59:39.132382 23974 raft_consensus.cc:993] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:39.231443 21258 rpcz_store.cc:276] 1215 08:59:37.694979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.710505 (+ 15526us) service_pool.cc:225] Handling call
1215 08:59:37.720957 (+ 10452us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:37.762456 (+ 41499us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:37.762487 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:37.779534 (+ 17047us) log.cc:844] Serialized 26 byte log entry
1215 08:59:37.992817 (+213283us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b1800463cc0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:39.121287 (+1128470us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:39.121327 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:39.121395 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:39.121437 (+ 42us) raft_consensus.cc:1685] finished
1215 08:59:39.121448 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:39.132075 (+ 10627us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":53324,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":33462,"spinlock_wait_cycles":9425792,"thread_start_us":1128298,"threads_started":1,"wal-append.queue_time_us":1279001}
I20251215 08:59:39.133018 23953 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.118574 23917 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20251215 08:59:39.141906 23900 raft_consensus.cc:3055] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:39.142356 21332 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:39.142764 23976 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.143177 23955 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.141914 21336 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:39.138257 21415 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:39.090642 21264 rpcz_store.cc:276] 1215 08:59:37.596746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.596931 (+ 185us) service_pool.cc:225] Handling call
1215 08:59:37.608518 (+ 11587us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:37.608787 (+ 269us) raft_consensus.cc:1534] Early marking committed up to index 9
1215 08:59:38.101046 (+492259us) log.cc:844] Serialized 14 byte log entry
1215 08:59:38.610919 (+509873us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:38.611000 (+ 81us) raft_consensus.cc:1645] Marking committed up to 9
1215 08:59:38.611083 (+ 83us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:38.611169 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:38.611329 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54866,"prepare.queue_time_us":900168,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":17012,"raft.queue_time_us":187868,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"thread_start_us":866742,"threads_started":2,"wal-append.queue_time_us":869985}
I20251215 08:59:39.143842 21977 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:39.150152 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.143446 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.117017 23874 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:39.086472 23925 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.179637 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.090953 23923 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.097067 23745 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.189493 21262 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:39.093197 23813 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.086727 23907 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.200433 21334 raft_consensus.cc:1275] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251215 08:59:39.131289 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.201889 21407 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:39.274291 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.274540 21407 raft_consensus.cc:606] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
W20251215 08:59:39.206974 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 53435.5us
I20251215 08:59:39.276688 21304 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:39.232890 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:39.239218 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:39.240808 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:39.279654 21411 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:39.090633 23964 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.153084 23979 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20251215 08:59:39.251664 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:39.247248 23708 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.247973 23900 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:39.259419 23837 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.259845 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:39.291245 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 53557.7us
I20251215 08:59:39.202198 23950 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.296449 23996 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:39.202008 23966 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.297051 23996 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.298688 23966 leader_election.cc:290] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:39.299183 23996 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:39.299332 23867 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.300351 21257 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:39.301779 21411 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:39.189280 23897 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.261574 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.261651 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.303185 21258 raft_consensus.cc:2410] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:39.268646 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.303915 21412 raft_consensus.cc:2410] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20251215 08:59:39.267932 21259 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:39.190084 23915 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.278853 21339 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:39.292281 23837 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.293948 23964 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.295523 23907 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.295885 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.091473 23960 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.324095 21972 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:39.324246 21262 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
I20251215 08:59:39.324712 23868 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.325129 23992 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:39.325266 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.325629 23960 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.325925 23981 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.325716 23992 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.183022 23983 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.326452 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:39.327034 21262 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:39.327917 23992 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.328368 23919 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.328850 23919 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.242149 23986 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.326445 21261 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:39.239352 23987 consensus_queue.cc:1048] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20251215 08:59:39.318310 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.313931 21264 raft_consensus.cc:2410] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251215 08:59:39.331066 23986 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.332016 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.332777 21264 raft_consensus.cc:2376] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7. Re-sending same reply.
I20251215 08:59:39.332636 23994 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:39.333716 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.334381 21264 raft_consensus.cc:2368] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:39.269666 23999 consensus_queue.cc:1059] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20251215 08:59:39.334591 23994 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:39.335054 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.329238 21414 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b83855234bda47a1b06c3f932728766b"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:39.336315 21414 raft_consensus.cc:606] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:39.336680 21414 raft_consensus.cc:614] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:39.258998 23953 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20251215 08:59:39.331022 23919 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.326911 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.332728 23955 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:39.349090 23848 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.350579 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:59:39.350914 21408 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:39.338120 21259 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:39.290807 23872 raft_consensus.cc:1081] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:39.328676 23708 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.302287 23765 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.358966 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.335073 23994 raft_consensus.cc:697] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:39.268842 23874 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.358439 21263 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:39.335186 23999 raft_consensus.cc:3055] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:39.366312 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.366570 21264 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:39.367117 21972 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:39.367866 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.370954 21259 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:39.327920 23983 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:39.346275 23889 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.374675 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.376993 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:39.387568 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:39.479233 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:39.486371 23837 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.486475 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.511145 21973 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:39.370929 23999 raft_consensus.cc:740] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:39.496944 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:39.513640 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 262039us
I20251215 08:59:39.388240 21264 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:39.505187 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.510047 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.527673 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:39.365837 23765 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:39.501796 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 225185us
I20251215 08:59:39.528584 21971 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:39.523327 21259 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:39.527385 21258 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:39.530210 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:39.496147 23925 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.527420 21262 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:39.504406 23992 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.533386 21979 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:39.533226 23992 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.534320 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:39.533133 23957 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
I20251215 08:59:39.535393 23992 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.536137 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.536743 21258 raft_consensus.cc:2376] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7. Re-sending same reply.
I20251215 08:59:39.537657 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.538794 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.539434 21258 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:39.540225 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.540776 21258 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:39.541682 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.542052 23874 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.542189 21258 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:39.543016 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.543936 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:39.546924 21307 leader_election.cc:336] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Timed out: RequestConsensusVote RPC to 127.19.163.65:46193 timed out after 1.615s (SENT)
W20251215 08:59:39.547351 23985 log.cc:927] Time spent T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.416s user 0.002s sys 0.000s
W20251215 08:59:39.547844 21307 leader_election.cc:336] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Timed out: RequestConsensusVote RPC to 127.19.163.65:46193 timed out after 1.646s (SENT)
W20251215 08:59:39.548694 21307 leader_election.cc:336] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Timed out: RequestConsensusVote RPC to 127.19.163.65:46193 timed out after 1.682s (SENT)
I20251215 08:59:39.549067 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:39.487675 23994 consensus_queue.cc:237] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.549324 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.544857 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:39.550078 21258 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 3
I20251215 08:59:39.550477 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.550678 23762 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.551025 21262 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:39.554569 23953 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:39.532145 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 271551us
I20251215 08:59:39.556708 21307 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:39.557471 23953 raft_consensus.cc:2804] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:39.557896 23953 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.557991 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:39.558295 23953 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:39.558794 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.559892 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.560447 21259 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:39.512930 23981 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.561321 23981 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.561703 23897 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:39.561841 23889 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:39.562970 23915 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.563249 23981 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.563932 21307 leader_election.cc:400] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:39.564198 21307 leader_election.cc:403] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:39.564332 23813 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.564646 23915 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:39.565548 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:39.557451 21264 raft_consensus.cc:1275] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:39.566407 21258 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:39.582940 21944 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:39.609422 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.609601 21230 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:39.610280 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.610585 23981 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:39.611169 23981 raft_consensus.cc:697] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:39.612135 23981 consensus_queue.cc:237] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.617537 21258 raft_consensus.cc:1275] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251215 08:59:39.618319 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 52838.7us
I20251215 08:59:39.619033 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:39.632740 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:39.601795 23867 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:39.634419 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:39.636893 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:39.638607 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 125432us
I20251215 08:59:39.624249 23889 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:39.601717 23917 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.560578 23900 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.624989 23762 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.602169 23966 consensus_queue.cc:1048] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.068s
I20251215 08:59:39.663933 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1777) took 1784 ms. Trace:
I20251215 08:59:39.647020 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1781) took 1626 ms. Trace:
I20251215 08:59:39.661134 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.645298 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:39.646458 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.684933 21979 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:39.667873 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:39.647403 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 64579.8us
I20251215 08:59:39.683805 21338 rpcz_store.cc:276] 1215 08:59:38.020653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.020816 (+ 163us) service_pool.cc:225] Handling call
1215 08:59:38.022806 (+ 1990us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:38.327764 (+304958us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:38.327791 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:38.327869 (+ 78us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:38.327898 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:39.646795 (+1318897us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b5400047c54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:39.646846 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:39.646972 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":700909440}
I20251215 08:59:39.663904 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1779) took 1779 ms. Trace:
I20251215 08:59:39.647929 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:39.687304 21335 rpcz_store.cc:276] 1215 08:59:37.884154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.884268 (+ 114us) service_pool.cc:225] Handling call
1215 08:59:37.958031 (+ 73763us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:38.321700 (+363669us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:38.321742 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:38.321791 (+ 49us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:38.321818 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:39.663625 (+1341807us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b54002819d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:39.663690 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:39.663836 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":835833216}
I20251215 08:59:39.681833 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.688979 21229 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:39.688069 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:39.683530 21337 rpcz_store.cc:276] 1215 08:59:37.879513 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.880049 (+ 536us) service_pool.cc:225] Handling call
1215 08:59:37.880455 (+ 406us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:37.965710 (+ 85255us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:39.121098 (+1155388us) log.cc:844] Serialized 14 byte log entry
1215 08:59:39.265091 (+143993us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:39.275365 (+ 10274us) log.cc:844] Serialized 52 byte log entry
1215 08:59:39.275521 (+ 156us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:39.275751 (+ 230us) log.cc:844] Serialized 14 byte log entry
1215 08:59:39.275959 (+ 208us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:39.276042 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:39.663719 (+387677us) raft_consensus.cc:1685] finished
1215 08:59:39.663742 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:39.663872 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37676,"prepare.queue_time_us":170384,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":92,"raft.queue_time_us":73925,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":15697,"spinlock_wait_cycles":26835712,"thread_start_us":1209040,"threads_started":2,"wal-append.queue_time_us":221674}
I20251215 08:59:39.685957 21974 raft_consensus.cc:2376] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7. Re-sending same reply.
I20251215 08:59:39.692947 21339 raft_consensus.cc:2393] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:39.664724 23874 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20251215 08:59:39.697568 21335 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:39.697818 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1502) took 1639 ms. Trace:
I20251215 08:59:39.670213 23923 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.638928 24008 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.700680 21410 rpcz_store.cc:276] 1215 08:59:38.058628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.058775 (+ 147us) service_pool.cc:225] Handling call
1215 08:59:38.059204 (+ 429us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:38.100351 (+ 41147us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:38.100393 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:39.551244 (+1450851us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b2c00092518. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:39.551630 (+ 386us) log.cc:844] Serialized 26 byte log entry
1215 08:59:39.631389 (+ 79759us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:39.631424 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:39.631492 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:39.697560 (+ 66068us) raft_consensus.cc:1685] finished
1215 08:59:39.697579 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:39.697761 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":218959,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":119501,"spinlock_wait_cycles":3890432,"thread_start_us":79401,"threads_started":1,"wal-append.queue_time_us":144176}
I20251215 08:59:39.701781 24008 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:39.702651 21332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 286) took 1437 ms (client timeout 1604 ms). Trace:
I20251215 08:59:39.699721 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:39.698318 23930 consensus_meta.cc:287] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.546s user 0.009s sys 0.003s
I20251215 08:59:39.703730 23930 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:39.703078 21332 rpcz_store.cc:269] 1215 08:59:38.265244 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.266520 (+ 1276us) service_pool.cc:225] Handling call
1215 08:59:39.702490 (+1435970us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54001243d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:39.702578 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":383336832}
I20251215 08:59:39.703830 24008 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.680169 23889 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.705260 21260 raft_consensus.cc:2368] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5.
I20251215 08:59:39.705324 23889 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:39.705909 23930 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.702294 23901 consensus_queue.cc:1059] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:39.707157 23900 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:39.670740 23745 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.638993 23955 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:39.708518 23955 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:39.708871 23955 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:39.708763 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.710647 21334 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:39.643896 21307 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:39.711364 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:39.712019 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 68247.4us
I20251215 08:59:39.712126 21258 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:39.712962 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:39.646143 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.701169 23896 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:39.665305 23994 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:39.714278 23896 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:39.714690 23896 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:39.714627 23994 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.662405 23869 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:39.646940 23987 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:39.639037 23986 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:39.718477 24008 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:39.754990 24008 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:39.755300 24008 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:39.756929 23994 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.757146 23986 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:39.757795 23986 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:39.690778 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:39.764506 24009 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.640156 23867 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.721431 23953 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.692842 23762 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.690105 23917 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.723886 23896 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.725528 24006 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:39.661491 23907 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:39.646515 23999 consensus_queue.cc:260] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.680809 23708 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.740322 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:39.752815 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 196180us
I20251215 08:59:39.765724 23847 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:39.719991 23975 raft_consensus.cc:3055] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:39.776175 23889 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.779054 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.804476 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.805202 21265 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:39.805413 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.806438 21414 raft_consensus.cc:2393] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:39.790725 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.793826 23847 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:39.796793 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.809167 21410 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:39.774747 23976 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:39.813555 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 73349.6us
W20251215 08:59:39.792181 23830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.022s sys 0.004s
I20251215 08:59:39.787537 23907 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:39.840428 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.792323 24006 raft_consensus.cc:697] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:39.784946 24009 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:39.938766 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:39.933458 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 287) took 1650 ms. Trace:
W20251215 08:59:39.939292 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:39.841368 21413 raft_consensus.cc:2410] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251215 08:59:39.946251 21336 rpcz_store.cc:276] 1215 08:59:38.282751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.301175 (+ 18424us) service_pool.cc:225] Handling call
1215 08:59:39.933233 (+1632058us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b5400115b54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:39.933396 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1917464960}
W20251215 08:59:39.840752 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 44033.3us
W20251215 08:59:39.947573 21307 leader_election.cc:336] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Timed out: RequestConsensusVote RPC to 127.19.163.65:46193 timed out after 1.550s (SENT)
I20251215 08:59:39.947793 21944 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:39.945595 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 116914us
I20251215 08:59:39.948576 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:39.797495 23975 raft_consensus.cc:740] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:39.840474 23830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.026s user 0.023s sys 0.004s
I20251215 08:59:39.970873 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.969103 23953 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.998364 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.999137 21258 raft_consensus.cc:2393] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:39.998961 21307 leader_election.cc:336] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193): Timed out: RequestConsensusVote RPC to 127.19.163.65:46193 timed out after 1.924s (SENT)
I20251215 08:59:39.999496 23864 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.000102 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.002266 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.003310 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.004017 21263 raft_consensus.cc:2393] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:40.003338 21975 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:39.973201 21414 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:39.968119 23868 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:39.973868 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:39.972199 23966 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:40.006453 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.006832 23966 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:40.007396 23966 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:40.007773 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.981647 24006 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.000741 21258 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:40.013531 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:39.983767 23889 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.016417 23879 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:39.862709 23907 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:39.918954 23896 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:39.923471 23867 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:39.920934 23917 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:40.022614 23956 log.cc:927] Time spent T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 1.689s user 0.006s sys 0.030s
W20251215 08:59:40.024902 21977 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 592) took 1713 ms (client timeout 1643 ms). Trace:
W20251215 08:59:40.025334 21977 rpcz_store.cc:269] 1215 08:59:38.311745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.311944 (+ 199us) service_pool.cc:225] Handling call
1215 08:59:40.024693 (+1712749us) spinlock_profiling.cc:243] Waited 61.1 ms on lock 0x7b540064d7d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:40.024806 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":140544384}
I20251215 08:59:40.027792 23981 raft_consensus.cc:3055] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:40.028188 23981 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:39.988948 23975 consensus_queue.cc:260] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:39.968169 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.032435 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1786) took 3028 ms. Trace:
I20251215 08:59:40.058830 21333 rpcz_store.cc:276] 1215 08:59:37.003822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.227711 (+223889us) service_pool.cc:225] Handling call
1215 08:59:37.618213 (+390502us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:37.618590 (+ 377us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:37.618614 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:37.619634 (+ 1020us) log.cc:844] Serialized 501 byte log entry
1215 08:59:38.044564 (+424930us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:38.044607 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:38.044673 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:40.022319 (+1977646us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b240001f2f8. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f398495a9a4 00007f3984958aab 00007f39849e79a9 00007f39849e7635 00007f39849e6c7d 00007f39877374ae 00007f398774ec2c 00007f398774df56 00007f398774db78 00007f39877575c1
1215 08:59:40.032113 (+ 9794us) raft_consensus.cc:1685] finished
1215 08:59:40.032137 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:40.032360 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316175616,"thread_start_us":424567,"threads_started":1,"wal-append.queue_time_us":636093}
W20251215 08:59:40.032871 21266 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 1951) took 2520 ms (client timeout 1589 ms). Trace:
I20251215 08:59:40.034938 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1465) took 1019 ms. Trace:
W20251215 08:59:40.037985 21408 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1505) took 1968 ms (client timeout 1500 ms). Trace:
W20251215 08:59:40.064122 21408 rpcz_store.cc:269] 1215 08:59:38.069730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.069860 (+ 130us) service_pool.cc:225] Handling call
1215 08:59:40.037751 (+1967891us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54002f53d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:40.037897 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2823953664}
I20251215 08:59:40.050158 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.066404 21971 raft_consensus.cc:2393] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
W20251215 08:59:40.061444 21266 rpcz_store.cc:269] 1215 08:59:37.512974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.534314 (+ 21340us) service_pool.cc:225] Handling call
1215 08:59:40.032665 (+2498351us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b54000cce54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:40.032786 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114375936}
I20251215 08:59:40.054416 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:40.133610 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 141085us
I20251215 08:59:40.135412 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:40.136406 21257 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:40.042052 21340 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:40.140189 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.134097 21263 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:40.142527 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:40.142987 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.062608 21260 rpcz_store.cc:276] 1215 08:59:39.015651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:39.015797 (+ 146us) service_pool.cc:225] Handling call
1215 08:59:40.034770 (+1018973us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b54002e4fd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:40.034890 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246214528}
I20251215 08:59:40.045563 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.147548 21337 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:40.148648 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.143077 21263 raft_consensus.cc:2376] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7. Re-sending same reply.
I20251215 08:59:40.153313 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.153386 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.153978 21263 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:40.154119 21262 raft_consensus.cc:2393] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:40.140921 21340 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
W20251215 08:59:40.156952 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 117002us
W20251215 08:59:40.067499 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 83258.2us
I20251215 08:59:40.159145 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.159716 21971 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:40.039399 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.160573 21413 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:40.160944 21378 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:40.044809 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.162640 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.164316 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.155850 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.165189 21257 raft_consensus.cc:2393] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:40.165537 21339 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:40.168862 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 120957us
I20251215 08:59:40.163604 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.036480 23918 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.035317 23911 raft_consensus.cc:3055] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:40.171209 21338 raft_consensus.cc:2393] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:40.203725 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.040166 23987 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:40.213151 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:40.214792 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.215457 21411 raft_consensus.cc:2368] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 2. Current term is 3.
W20251215 08:59:40.216074 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 56312.9us
I20251215 08:59:40.185056 21339 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20251215 08:59:40.216780 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 55908.9us
I20251215 08:59:40.187273 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.216948 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.041164 23765 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.194319 21332 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251215 08:59:40.184423 21335 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:40.179853 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:39.973533 23847 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.234802 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:39.994544 24009 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.216150 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.217901 21407 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:40.171761 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.253641 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.029764 23975 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:40.241582 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.257916 21415 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:40.241849 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.242452 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.241582 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:40.247375 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.249470 21336 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:40.254412 21339 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:40.141175 23955 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.211273 23911 raft_consensus.cc:740] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:40.262389 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 328516us
I20251215 08:59:40.260936 21976 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:40.249655 23879 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:40.280925 21381 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:40.281553 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:40.271032 21339 raft_consensus.cc:1275] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:40.227008 23965 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:39.992624 24008 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:40.276562 21232 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.714s (SENT) [suppressed 36 similar messages]
I20251215 08:59:40.289798 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.293147 21381 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:40.279197 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:40.262159 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.314381 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.314963 21407 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:40.281025 21334 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:40.269529 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.315593 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.288179 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:40.289758 21341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1790) took 2557 ms (client timeout 1813 ms). Trace:
I20251215 08:59:40.028990 23981 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.316164 21332 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:40.280674 23879 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:40.342463 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1965) took 1139 ms. Trace:
W20251215 08:59:40.343729 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 62247.3us
I20251215 08:59:40.261694 23966 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.323312 21411 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251215 08:59:40.353724 23762 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.357066 21381 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:40.361641 21408 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
W20251215 08:59:40.338403 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 59296.9us
I20251215 08:59:40.349839 21259 rpcz_store.cc:276] 1215 08:59:39.203287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:39.560944 (+357657us) service_pool.cc:225] Handling call
1215 08:59:40.046724 (+485780us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:40.153861 (+107137us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:40.153887 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:40.153928 (+ 41us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:40.153954 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:40.154014 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:40.342293 (+188279us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54000cce50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:40.342398 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113626,"raft.queue_time_us":107255,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":46058,"spinlock_wait_cycles":1116521344}
I20251215 08:59:40.363927 21946 leader_election.cc:304] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:40.332847 21415 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20251215 08:59:40.318527 21341 rpcz_store.cc:269] 1215 08:59:37.722861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.722975 (+ 114us) service_pool.cc:225] Handling call
1215 08:59:40.280486 (+2557511us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400048654. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:40.280624 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":516256512}
I20251215 08:59:40.368188 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.335614 21307 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:40.282588 24003 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:40.319451 23901 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:40.370822 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.388729 21411 raft_consensus.cc:2410] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251215 08:59:40.301841 23999 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:40.402806 21407 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:40.371340 23879 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:40.370537 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:40.443495 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 73029.4us
I20251215 08:59:40.444015 21407 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251215 08:59:40.378671 21413 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:40.445570 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.446856 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.447475 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.447829 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.447947 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.448169 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.448768 21260 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:40.449596 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.371436 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.313424 23984 consensus_queue.cc:1059] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.073s
I20251215 08:59:40.450377 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:40.375931 21307 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:40.317492 23932 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:40.350323 23988 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:40.338913 23986 consensus_meta.cc:287] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.581s user 0.005s sys 0.004s
I20251215 08:59:40.321424 23800 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:40.314782 23897 consensus_queue.cc:1048] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.130s
I20251215 08:59:40.302466 23965 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.353823 23911 consensus_queue.cc:260] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:40.451112 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:40.458194 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 82350.3us
I20251215 08:59:40.374928 23987 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:40.374348 23900 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:40.343010 23861 raft_consensus.cc:2804] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:40.475980 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.311801 23750 consensus_queue.cc:1059] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.062s
I20251215 08:59:40.449717 23813 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:40.395207 23974 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:40.364256 23892 consensus_queue.cc:1048] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
W20251215 08:59:40.518954 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:40.359223 23907 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.527474 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:40.540778 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:40.544523 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:40.508095 21261 raft_consensus.cc:1240] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:40.457859 23932 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:40.559767 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:40.479511 23988 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:40.564628 21411 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:40.568126 21944 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:40.570672 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.528117 23987 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:40.627637 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.512975 23992 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:40.629691 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.632665 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:40.635641 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:40.439890 23901 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:40.570127 23917 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:40.642120 23917 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:40.642525 23917 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:40.646936 23901 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.647121 23907 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:40.649539 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1969) took 1445 ms. Trace:
I20251215 08:59:40.650074 21264 rpcz_store.cc:276] 1215 08:59:39.204199 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:39.611459 (+407260us) service_pool.cc:225] Handling call
1215 08:59:39.741357 (+129898us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:39.741568 (+ 211us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:39.741603 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:39.741657 (+ 54us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:39.741685 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:39.741747 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:40.649330 (+907583us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b5400289e50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:40.649467 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":297880960}
I20251215 08:59:40.651681 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.653682 23966 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:40.654757 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:40.640525 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:40.655368 23911 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:40.630867 23900 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:40.656002 23900 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:40.656042 23830 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:40.656579 23900 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:40.657194 23967 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:40.657836 23967 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.660372 23967 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:40.665288 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 983453us
I20251215 08:59:40.665628 21973 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:40.629431 23987 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:40.669334 23635 raft_consensus.cc:993] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:40.656710 23830 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:40.672427 23830 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:40.632696 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 64671.2us
I20251215 08:59:40.671857 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1725) took 2954 ms. Trace:
I20251215 08:59:40.675338 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.675891 21409 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:40.674616 21406 rpcz_store.cc:276] 1215 08:59:37.717407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:37.717800 (+ 393us) service_pool.cc:225] Handling call
1215 08:59:37.729415 (+ 11615us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:37.881914 (+152499us) raft_consensus.cc:1534] Early marking committed up to index 7
1215 08:59:38.237994 (+356080us) log.cc:844] Serialized 14 byte log entry
1215 08:59:39.122842 (+884848us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:39.122892 (+ 50us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:39.131220 (+ 8328us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:40.671050 (+1539830us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b1800472720. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ad3fe6 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49
1215 08:59:40.671607 (+ 557us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540025fb54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:40.671653 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:40.671776 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1913,"prepare.queue_time_us":781511,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":6283,"raft.queue_time_us":2070,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":350200064,"thread_start_us":1118182,"threads_started":2,"wal-append.queue_time_us":713772,"wal-append.run_cpu_time_us":1135,"wal-append.run_wall_time_us":1009579}
I20251215 08:59:40.672765 23992 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.685181 23873 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:40.685669 23873 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:40.687469 23873 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.680792 23861 raft_consensus.cc:697] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:40.691883 23861 consensus_queue.cc:237] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:40.692842 23965 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:40.563999 23997 log.cc:927] Time spent T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20251215 08:59:40.696782 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.454427 24003 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:40.570516 23932 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:40.553544 23974 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:40.717475 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.654986 23917 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:40.506697 23986 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.707962 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:40.726987 21259 raft_consensus.cc:2376] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7. Re-sending same reply.
I20251215 08:59:40.564450 24008 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.543584 23955 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.718413 21409 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:40.723209 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.638345 23988 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:40.515069 23800 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.699242 23992 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.704552 23989 raft_consensus.cc:1081] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:40.704039 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.736747 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.740867 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:40.741451 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:40.725661 23851 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.629s user 0.010s sys 0.026s
I20251215 08:59:40.747215 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.740929 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.725176 23974 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:40.741477 21413 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:40.762723 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:40.747351 23851 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.650s user 0.010s sys 0.027s
W20251215 08:59:40.741937 21975 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.018s sys 0.027s
I20251215 08:59:40.735001 23745 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:40.767673 21975 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.019s sys 0.027s
I20251215 08:59:40.768499 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 597) took 1786 ms. Trace:
I20251215 08:59:40.773279 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:40.738188 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:40.769902 21978 rpcz_store.cc:276] 1215 08:59:38.982061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:38.982561 (+ 500us) service_pool.cc:225] Handling call
1215 08:59:38.982885 (+ 324us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:39.296041 (+313156us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:39.296074 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:39.296126 (+ 52us) raft_consensus.cc:1645] Marking committed up to 9
1215 08:59:39.296162 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:40.768198 (+1472036us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54003ec954. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:40.768259 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:40.768429 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":719704960}
I20251215 08:59:40.786276 21414 raft_consensus.cc:1240] T 60989b82e8354098aac2066e31697953 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 9. Ops: []
I20251215 08:59:40.791579 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.788305 23976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32792 (request call id 850) took 5354 ms. Trace:
I20251215 08:59:40.773669 23762 consensus_queue.cc:1059] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20251215 08:59:40.809548 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.777704 23745 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.817126 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:40.819490 21977 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:40.826304 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.827805 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.829036 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.829416 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.830160 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.830644 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.831348 21406 raft_consensus.cc:2368] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5.
I20251215 08:59:40.834527 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.835042 21409 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:40.837649 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.838547 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.881700 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:40.882395 21406 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:40.837550 23846 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:40.886508 23986 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.841487 23979 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:40.899763 23846 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.813313 24006 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:40.802865 23976 rpcz_store.cc:276] 1215 08:59:35.437655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:35.443967 (+ 6312us) service_pool.cc:225] Handling call
1215 08:59:35.673268 (+229301us) log.cc:844] Serialized 501 byte log entry
1215 08:59:36.102626 (+429358us) spinlock_profiling.cc:243] Waited 6.35 ms on lock 0x7b18000d6c00. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ad3fe6 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987b34b56 00007f3987b2e4ca 00007f3987b2d744
1215 08:59:36.102999 (+ 373us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b5400235da0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987b34b56 00007f3987b2e4ca 00007f3987b2d744 00007f3987b0deda 00007f3987b14d83 00007f3987b14d09 00007f3987b14c99 00007f3987b14c61 00007f3987b13e2d 0000000000379254
1215 08:59:36.374807 (+271808us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:36.386817 (+ 12010us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b5400235da0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987b2d172 00007f3987b0deda 00007f3987b14d83 00007f3987b14d09 00007f3987b14c99 00007f3987b14c61 00007f3987b13e2d 0000000000379254 0000000000379099 00007f3984b06065
1215 08:59:40.774446 (+4387629us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1604,"raft.run_cpu_time_us":5587,"raft.run_wall_time_us":967656,"spinlock_wait_cycles":41601920,"thread_start_us":63453,"threads_started":1,"wal-append.queue_time_us":79872,"wal-append.run_cpu_time_us":6079,"wal-append.run_wall_time_us":1778446}
I20251215 08:59:40.896318 21406 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
W20251215 08:59:40.898313 21412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.019s sys 0.022s
I20251215 08:59:40.913424 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:40.903822 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:40.912891 21412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.019s sys 0.022s
I20251215 08:59:40.903652 23846 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.900236 23979 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.905738 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:40.920537 21409 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:40.921581 24029 log.cc:927] Time spent T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20251215 08:59:40.922148 24003 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.925559 23987 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:40.981177 23987 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.981263 23979 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.913998 23879 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.983690 23879 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:40.912149 23917 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.984812 21407 raft_consensus.cc:2368] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6.
I20251215 08:59:40.920926 23900 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.986920 23900 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:40.929247 23830 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.934715 24007 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:40.936347 23992 raft_consensus.cc:3055] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Stepping down as leader of term 8
I20251215 08:59:40.988802 23992 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:40.989244 23830 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.989639 23992 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:40.990957 23992 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:40.937254 21979 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:40.926753 23745 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:40.990159 21976 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:40.994169 21975 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:41.008013 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:41.008204 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:41.008846 21979 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.012974 21406 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:40.994573 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 101929us
W20251215 08:59:40.950733 21971 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.000s sys 0.018s
I20251215 08:59:41.017206 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:41.017653 21971 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.000s sys 0.019s
I20251215 08:59:41.017838 21411 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:41.000567 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:41.002430 23999 consensus_meta.cc:287] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.560s user 0.005s sys 0.005s
W20251215 08:59:41.003487 23975 consensus_meta.cc:287] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.743s user 0.002s sys 0.008s
I20251215 08:59:41.018683 21409 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:41.038414 21410 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:41.036273 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:41.042616 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:41.042510 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.025125 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:41.034852 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:41.046233 21339 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:41.037065 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:41.042943 21381 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:41.048146 21336 raft_consensus.cc:1275] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:41.040665 21980 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:41.054111 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:41.073751 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:41.074759 21333 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
W20251215 08:59:41.077910 21257 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.013s sys 0.018s
W20251215 08:59:41.079231 21257 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.509s user 0.013s sys 0.018s
W20251215 08:59:41.086120 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:41.086949 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:40.946271 23988 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.087074 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.036540 23987 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:41.146061 21414 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:41.146617 21332 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:41.087531 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.116712 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:41.147568 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 58390.2us
I20251215 08:59:41.151612 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:41.093214 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 50367.6us
W20251215 08:59:41.148684 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 98534.7us
I20251215 08:59:41.156970 21381 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.019876 23800 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:41.158085 21409 raft_consensus.cc:2376] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5. Re-sending same reply.
W20251215 08:59:41.160061 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:41.161731 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.063591 23996 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:41.167613 21336 raft_consensus.cc:1240] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:41.162632 21409 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:41.167913 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:41.064479 23986 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.171967 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:41.203792 21230 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:41.170941 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:41.187740 21232 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:41.201684 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 44835.2us
I20251215 08:59:41.233088 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:41.185153 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.180655 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1502) took 1127 ms. Trace:
I20251215 08:59:41.050910 24009 consensus_queue.cc:1059] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.122s
I20251215 08:59:41.236195 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:41.247285 21266 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:41.219842 21946 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:41.198493 21415 raft_consensus.cc:1240] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:41.200033 21336 raft_consensus.cc:1240] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:41.145074 23851 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:41.148555 23837 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.245553 21230 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.259549 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:41.238956 21258 rpcz_store.cc:276] 1215 08:59:40.053211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.158829 (+105618us) service_pool.cc:225] Handling call
1215 08:59:40.209069 (+ 50240us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:40.240025 (+ 30956us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:40.240095 (+ 70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:40.256822 (+ 16727us) log.cc:844] Serialized 26 byte log entry
1215 08:59:40.736815 (+479993us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:41.178920 (+442105us) log.cc:844] Serialized 14 byte log entry
1215 08:59:41.179824 (+ 904us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:41.180318 (+ 494us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:41.180379 (+ 61us) raft_consensus.cc:1685] finished
1215 08:59:41.180398 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:41.180563 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":200,"prepare.queue_time_us":144335,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":29,"raft.queue_time_us":10117,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":42564,"thread_start_us":837110,"threads_started":2,"wal-append.queue_time_us":517957}
W20251215 08:59:41.261461 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 187763us
I20251215 08:59:41.264370 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:41.265452 21334 raft_consensus.cc:2393] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:41.266014 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 46237.4us
I20251215 08:59:41.267203 21946 leader_election.cc:400] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5.
I20251215 08:59:41.267642 21946 leader_election.cc:403] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251215 08:59:41.257483 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:41.271220 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:41.272014 21336 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:41.214998 23911 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:41.152789 23901 consensus_queue.cc:1059] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.112s
I20251215 08:59:41.290935 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:41.244177 23955 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.288193 23978 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:41.296757 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:41.165206 23976 consensus_queue.cc:1059] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.125s
I20251215 08:59:41.303023 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:41.282960 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:41.317680 21946 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:41.322448 21977 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:41.284328 21332 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
W20251215 08:59:41.323712 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 84960.7us
I20251215 08:59:41.278385 23984 raft_consensus.cc:3055] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:41.189271 23996 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.278854 23999 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:41.330341 21307 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:41.332140 21332 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:41.254292 23896 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.047s
I20251215 08:59:41.283936 24037 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:41.325755 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:41.335666 23708 raft_consensus.cc:3055] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:41.313040 23978 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.328158 23851 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:41.255657 23986 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.325682 21946 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:41.323355 24033 log.cc:927] Time spent T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20251215 08:59:41.342271 21332 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:41.336426 24043 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:41.312753 23917 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:41.323457 21151 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 7edcdb5749ae4e59ab17648fe97390e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251215 08:59:41.295044 23762 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:41.349922 24037 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.160116 23987 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.328940 24003 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:41.335799 23988 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:41.338430 23974 consensus_meta.cc:287] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.569s user 0.003s sys 0.006s
I20251215 08:59:41.391279 23987 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:41.391209 23974 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.392086 21304 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.392374 24042 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:41.393216 21336 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:41.393748 21336 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:41.393833 23974 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:41.292289 23911 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:41.394544 23911 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:41.394609 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:41.328464 23983 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:41.341753 24038 raft_consensus.cc:3055] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:41.333803 24005 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5.
I20251215 08:59:41.366561 21381 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.333945 23984 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:41.272776 23837 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.367753 21946 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.342111 23750 raft_consensus.cc:3055] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:41.317147 23955 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.421490 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:41.355436 23932 consensus_meta.cc:287] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.629s user 0.005s sys 0.005s
W20251215 08:59:41.379423 21151 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 364161us
I20251215 08:59:41.353204 23708 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:41.359750 23978 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:41.381181 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 121732us
I20251215 08:59:41.400163 23983 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:41.439648 21339 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:41.440256 21230 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:41.385854 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:41.388417 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 140307us
I20251215 08:59:41.441862 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:41.390275 24036 log.cc:927] Time spent T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20251215 08:59:41.394119 21336 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.401038 24038 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:41.327764 24039 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:41.368641 23999 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:41.398515 21333 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:41.429826 23830 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:41.485692 23830 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:41.486171 23830 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:41.375922 24043 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.380759 23917 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.404946 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:41.411314 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:41.368520 23851 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.414825 23900 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:41.496280 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 91396.8us
I20251215 08:59:41.417150 21265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 293) took 1724 ms. Trace:
I20251215 08:59:41.406054 23750 raft_consensus.cc:740] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:41.427418 21979 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:41.395928 24003 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.444034 21262 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:41.396364 21408 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:41.498369 21378 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.475353 23983 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.449258 24041 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:41.449926 23992 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:41.448920 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:41.453701 23896 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:41.500572 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 51723.7us
W20251215 08:59:41.455911 24026 log.cc:927] Time spent T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.722s user 0.004s sys 0.000s
I20251215 08:59:41.456984 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1743) took 1184 ms. Trace:
I20251215 08:59:41.458761 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1526) took 1500 ms. Trace:
I20251215 08:59:41.510535 21410 rpcz_store.cc:276] 1215 08:59:39.958115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:39.958267 (+ 152us) service_pool.cc:225] Handling call
1215 08:59:41.458592 (+1500325us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54002f62d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:41.458700 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2274045184}
I20251215 08:59:41.009101 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:41.464890 23984 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.467162 21266 raft_consensus.cc:1275] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:41.467885 23708 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.471511 23981 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:41.472173 23879 raft_consensus.cc:3055] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:41.501310 23999 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:41.467885 23996 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:41.473433 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:41.506815 24039 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:41.478727 21980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46170 (request call id 483) took 1834 ms (client timeout 1961 ms). Trace:
I20251215 08:59:41.396934 21304 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.489243 21381 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:41.527405 21980 rpcz_store.cc:269] 1215 08:59:39.644390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:39.645687 (+ 1297us) service_pool.cc:225] Handling call
1215 08:59:41.478581 (+1832894us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003f02d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:41.478673 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3116332928}
I20251215 08:59:41.416527 23837 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.497071 21265 rpcz_store.cc:276] 1215 08:59:39.692277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:39.805941 (+113664us) service_pool.cc:225] Handling call
1215 08:59:39.945585 (+139644us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:39.949131 (+ 3546us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:39.949159 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:39.970311 (+ 21152us) log.cc:844] Serialized 26 byte log entry
1215 08:59:40.339984 (+369673us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b18004df6e0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:40.808898 (+468914us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:40.808960 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:40.809073 (+ 113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:41.402782 (+593709us) raft_consensus.cc:1685] finished
1215 08:59:41.402809 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:41.417080 (+ 14271us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91158,"raft.queue_time_us":1179,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":8587,"spinlock_wait_cycles":122550016,"thread_start_us":467480,"threads_started":1,"wal-append.queue_time_us":1152202}
I20251215 08:59:41.516053 23984 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:41.522492 24006 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:41.507827 21230 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.439617 23932 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.497694 24003 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:41.509858 21415 rpcz_store.cc:276] 1215 08:59:40.272986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.371825 (+ 98839us) service_pool.cc:225] Handling call
1215 08:59:40.437497 (+ 65672us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:41.251271 (+813774us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:41.456738 (+205467us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b5400341a54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:41.456920 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1749640576}
I20251215 08:59:41.515017 21975 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:41.544039 21230 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:41.497838 23917 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:41.496044 24042 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:41.496455 24043 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:41.468819 21160 catalog_manager.cc:5654] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 2 to 4, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 4 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:41.521908 23900 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:41.407277 23975 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:41.506676 23861 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:41.561789 23900 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:41.528028 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 131170us
I20251215 08:59:41.504206 23896 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.581432 23896 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.581624 23955 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.582325 23955 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.582379 21304 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.582859 23955 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.583674 21304 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.507588 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 300) took 1210 ms. Trace:
I20251215 08:59:41.534865 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:41.537729 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 207455us
I20251215 08:59:41.534382 21336 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.518671 23879 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:41.546001 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:41.548511 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:41.530642 23708 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:41.531001 24039 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.531771 23846 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251215 08:59:41.529783 24007 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:41.532949 23999 raft_consensus.cc:697] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:41.536487 24037 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.541474 24006 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.516268 24038 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.543713 23986 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:41.586944 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 662) took 1139 ms. Trace:
I20251215 08:59:41.600371 21304 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.584873 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:41.562536 21974 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:41.588020 24042 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
W20251215 08:59:41.606906 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 61020.2us
I20251215 08:59:41.569710 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1752) took 1017 ms. Trace:
I20251215 08:59:41.625147 21408 rpcz_store.cc:276] 1215 08:59:40.552819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.553647 (+ 828us) service_pool.cc:225] Handling call
1215 08:59:41.569511 (+1015864us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54001761d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:41.569651 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1754094976}
I20251215 08:59:41.626096 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:41.611632 23996 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:41.577970 21332 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.610932 21412 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:41.614675 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 427062us
I20251215 08:59:41.605738 21335 rpcz_store.cc:276] 1215 08:59:40.297269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.299044 (+ 1775us) service_pool.cc:225] Handling call
1215 08:59:40.320202 (+ 21158us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:40.320526 (+ 324us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:40.320551 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:40.331208 (+ 10657us) log.cc:844] Serialized 52 byte log entry
1215 08:59:40.672612 (+341404us) spinlock_profiling.cc:243] Waited 7.16 ms on lock 0x7b18002033e0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:41.033118 (+360506us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:41.313172 (+280054us) log.cc:844] Serialized 14 byte log entry
1215 08:59:41.493930 (+180758us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:41.494001 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:41.494042 (+ 41us) raft_consensus.cc:1685] finished
1215 08:59:41.507394 (+ 13352us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:41.507533 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54613,"prepare.queue_time_us":119546,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":3714,"raft.queue_time_us":72258,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":16739,"spinlock_wait_cycles":16460160,"thread_start_us":688740,"threads_started":3,"wal-append.queue_time_us":1015573}
W20251215 08:59:41.617486 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 82728.8us
I20251215 08:59:41.633692 21381 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.634879 21332 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.555630 24041 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:41.584825 23955 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.556885 23851 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:41.563485 23918 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:41.562842 23901 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:41.623384 21971 rpcz_store.cc:276] 1215 08:59:40.447407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.448754 (+ 1347us) service_pool.cc:225] Handling call
1215 08:59:40.449058 (+ 304us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:41.030264 (+581206us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:41.030314 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:41.078996 (+ 48682us) log.cc:844] Serialized 26 byte log entry
1215 08:59:41.410603 (+331607us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:41.410635 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:41.410695 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:41.586731 (+176036us) raft_consensus.cc:1685] finished
1215 08:59:41.586751 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:41.586884 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30606,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":19419,"thread_start_us":778592,"threads_started":2,"wal-append.queue_time_us":386347}
I20251215 08:59:41.628012 21304 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.605842 21265 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:41.628286 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.609449 21336 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.621965 23959 consensus_queue.cc:1059] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
W20251215 08:59:41.609592 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:41.628890 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 44090.7us
I20251215 08:59:41.618494 23861 raft_consensus.cc:697] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:41.614459 24007 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.654819 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.611862 24039 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.621137 23624 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:41.629679 23987 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:41.578303 23837 consensus_queue.cc:1048] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20251215 08:59:41.655892 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:41.643052 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:41.684623 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:41.680552 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.709333 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:41.636567 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:41.710309 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.627351 24061 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:41.639824 23872 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:41.650842 23901 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:41.699000 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:41.699465 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 71274.1us
I20251215 08:59:41.686980 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.668324 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.701697 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.753235 21411 raft_consensus.cc:2410] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251215 08:59:41.702032 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:41.637848 23975 consensus_queue.cc:1048] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20251215 08:59:41.638440 24042 raft_consensus.cc:697] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:41.703287 21415 raft_consensus.cc:1275] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:41.586971 23932 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.711247 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.713721 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.648165 24063 raft_consensus.cc:2764] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251215 08:59:41.754048 21413 raft_consensus.cc:2376] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7. Re-sending same reply.
I20251215 08:59:41.706321 23996 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:41.759691 21408 raft_consensus.cc:2376] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5. Re-sending same reply.
I20251215 08:59:41.741447 23955 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:41.759670 21946 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.761343 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.792194 21411 raft_consensus.cc:2393] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:41.758989 21414 raft_consensus.cc:2376] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7. Re-sending same reply.
W20251215 08:59:41.758275 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 124643us
I20251215 08:59:41.698473 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:41.629689 23999 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.787556 21407 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 4. Current term is 5. Ops: []
W20251215 08:59:41.788846 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 160875us
I20251215 08:59:41.847062 23987 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:41.847303 21304 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.847610 23987 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:41.848073 23987 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:41.849161 24059 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:41.791263 21413 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
W20251215 08:59:41.798936 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:41.785847 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 74729.3us
I20251215 08:59:41.850015 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:41.736368 23918 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:41.759943 23901 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:41.899463 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 201027us
I20251215 08:59:41.891270 21409 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:41.898828 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 191275us
W20251215 08:59:41.897608 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 46926.3us
I20251215 08:59:41.903900 21332 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:41.904757 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.894781 21410 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:41.907179 21307 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:41.909128 21307 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:41.909540 21334 raft_consensus.cc:1275] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:41.917888 21232 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:41.920879 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:41.641178 23879 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:41.861311 24047 log.cc:927] Time spent T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20251215 08:59:41.788678 23996 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:41.871131 23918 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.943528 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:41.855792 23955 raft_consensus.cc:697] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:41.791180 23872 consensus_queue.cc:1059] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.146s
I20251215 08:59:41.945106 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:41.947474 21381 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:41.948706 21332 raft_consensus.cc:2376] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5. Re-sending same reply.
I20251215 08:59:41.954595 21335 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:41.896651 23932 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:41.865047 24061 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:41.935120 23911 consensus_meta.cc:287] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.540s user 0.008s sys 0.004s
I20251215 08:59:41.920529 24060 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:41.864967 24057 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20251215 08:59:41.865123 23992 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
W20251215 08:59:41.963155 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 203579us
W20251215 08:59:41.964355 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:41.892694 23978 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:41.955438 23932 raft_consensus.cc:697] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:41.976027 21946 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:41.951404 23837 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:41.983404 23837 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:41.983848 23837 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:41.983616 23932 consensus_queue.cc:237] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.949877 23996 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:41.972565 23911 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.985378 23955 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.986634 23911 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:41.987702 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:41.969508 24061 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:41.988619 24061 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:41.995743 21263 raft_consensus.cc:2410] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20251215 08:59:41.997108 21263 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 1980) took 1551 ms (client timeout 2037 ms). Trace:
W20251215 08:59:41.997470 21263 rpcz_store.cc:269] 1215 08:59:40.446025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.449383 (+ 3358us) service_pool.cc:225] Handling call
1215 08:59:41.996894 (+1547511us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54001342d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:41.997044 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3553848064}
I20251215 08:59:41.998534 23762 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:41.999073 23762 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.999591 23750 consensus_queue.cc:260] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.000633 23750 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:41.864792 24059 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.002962 24059 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:41.921551 24058 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20251215 08:59:41.981436 21266 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:41.982583 23978 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.008590 21974 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:42.009379 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:42.009409 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:42.010569 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.011121 24075 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:42.008642 24007 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.012190 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:41.940593 23830 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.013890 21975 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.014425 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:42.014900 23959 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:42.015030 21410 pending_rounds.cc:85] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Aborting all ops after (but not including) 8
I20251215 08:59:42.046250 21410 pending_rounds.cc:107] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20251215 08:59:42.046633 21410 raft_consensus.cc:2889] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:42.015509 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.016376 21415 raft_consensus.cc:1240] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:42.017740 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:42.017587 23996 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:42.019057 24050 log.cc:927] Time spent T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.541s user 0.008s sys 0.011s
I20251215 08:59:42.019520 24039 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.018224 23987 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.025842 24064 raft_consensus.cc:3055] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Stepping down as leader of term 8
I20251215 08:59:41.953318 24070 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:42.012637 21262 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:42.036214 21381 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:42.071020 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 153248us
I20251215 08:59:42.073160 21381 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.073417 21232 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:41.893146 23861 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:41.961719 24072 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:42.008705 23986 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:42.033295 24075 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:42.062412 21406 raft_consensus.cc:2393] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:42.004644 23900 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.020641 23837 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.085517 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.080302 21978 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 8: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:42.015537 23896 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.105253 21257 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.105403 24080 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:42.104501 21232 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.064188 23999 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:42.106055 23896 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:42.097460 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 71477.5us
I20251215 08:59:42.107275 23896 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:42.108326 21232 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:42.109277 23896 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:42.109769 23896 raft_consensus.cc:697] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:42.110168 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 259) took 1282 ms. Trace:
I20251215 08:59:42.110541 21412 rpcz_store.cc:276] 1215 08:59:40.827814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.915768 (+ 87954us) service_pool.cc:225] Handling call
1215 08:59:41.570468 (+654700us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:41.632351 (+ 61883us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:42.109297 (+476946us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b54001761d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:42.109986 (+ 689us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54001761d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:42.110113 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1597464704}
I20251215 08:59:42.110636 23896 consensus_queue.cc:237] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.114233 24080 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:42.116770 24080 consensus_queue.cc:1059] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:42.118208 23981 raft_consensus.cc:3055] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:42.118623 23981 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:42.118399 23974 consensus_queue.cc:1059] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:42.119565 23974 raft_consensus.cc:3055] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:42.119457 23981 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.119925 23974 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.121137 23981 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:42.120985 23974 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.123049 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:42.124378 23700 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:42.124744 23700 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:42.125084 23700 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.129338 23974 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.129952 21232 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.132872 24080 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:42.134471 21232 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:42.135006 23981 consensus_queue.cc:1048] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:42.108654 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.070648 24064 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.138187 21232 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.079345 23872 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:42.071724 24070 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:42.148442 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.147935 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.088662 23978 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:42.151469 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.156422 21232 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:42.099526 24075 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.089305 24073 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:42.158272 21232 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:42.059269 23959 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.137483 23762 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.148000 24080 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election won for term 7
W20251215 08:59:42.184445 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 173932us
I20251215 08:59:42.151065 23992 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251215 08:59:42.186800 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 277781us
W20251215 08:59:42.198738 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 43777.8us
I20251215 08:59:42.193259 21946 leader_election.cc:400] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6.
I20251215 08:59:42.199746 21946 leader_election.cc:403] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251215 08:59:42.158360 23976 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:42.160069 24059 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:42.099829 23986 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.153904 24070 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:42.154412 23983 consensus_meta.cc:287] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.648s user 0.014s sys 0.006s
I20251215 08:59:42.203155 21973 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.160454 24060 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:42.158349 23918 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.198206 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.239284 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.206872 23976 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:42.207437 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:41.979517 24042 consensus_queue.cc:237] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.179927 23965 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251215 08:59:42.192108 24043 consensus_meta.cc:287] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.597s user 0.005s sys 0.015s
I20251215 08:59:42.211079 21257 raft_consensus.cc:1240] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:42.212457 23837 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.212862 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.214637 23872 consensus_queue.cc:1059] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:42.216436 23959 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.182482 24058 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:42.220829 24007 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:42.221807 21409 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:42.222357 23996 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.157703 24068 consensus_queue.cc:1048] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
W20251215 08:59:42.226362 24003 consensus_meta.cc:287] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.669s user 0.008s sys 0.001s
I20251215 08:59:42.208036 24062 consensus_queue.cc:1059] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
W20251215 08:59:42.227568 23984 consensus_meta.cc:287] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.694s user 0.017s sys 0.021s
I20251215 08:59:42.208985 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1993) took 1311 ms. Trace:
I20251215 08:59:42.199106 23762 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:42.231901 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:42.261251 23846 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:42.237066 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:42.284376 21262 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.244297 21266 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.184069 23975 consensus_queue.cc:1059] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20251215 08:59:42.248445 21379 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.233050 24080 raft_consensus.cc:697] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:42.288985 23975 raft_consensus.cc:3055] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:42.289469 23975 raft_consensus.cc:740] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.289829 24080 consensus_queue.cc:237] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.290324 23975 consensus_queue.cc:260] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.290671 23901 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.248805 24060 raft_consensus.cc:697] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:42.249115 24063 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20251215 08:59:42.247895 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:42.251192 23872 raft_consensus.cc:3055] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:42.254567 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:42.256088 23837 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:42.256551 23986 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.210521 23999 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6.
I20251215 08:59:42.249648 24043 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.247264 24089 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:42.204089 23624 consensus_queue.cc:1059] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20251215 08:59:42.246182 23976 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:42.251094 24039 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.281751 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:42.282325 21263 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.319857 24089 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:42.321506 23624 raft_consensus.cc:3055] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:42.321839 23624 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:42.322147 23917 consensus_meta.cc:287] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.714s user 0.019s sys 0.000s
I20251215 08:59:42.322515 24068 consensus_queue.cc:1059] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:42.322576 23624 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.284703 23830 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.148451 23700 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.281999 21264 rpcz_store.cc:276] 1215 08:59:40.897785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:40.898065 (+ 280us) service_pool.cc:225] Handling call
1215 08:59:40.898393 (+ 328us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:40.898757 (+ 364us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:40.898783 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:40.913195 (+ 14412us) log.cc:844] Serialized 52 byte log entry
1215 08:59:41.226710 (+313515us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:41.265512 (+ 38802us) log.cc:844] Serialized 14 byte log entry
1215 08:59:41.531287 (+265775us) log.cc:844] Serialized 14 byte log entry
1215 08:59:41.531568 (+ 281us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:41.532617 (+ 1049us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:42.018527 (+485910us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:42.208711 (+190184us) raft_consensus.cc:1685] finished
1215 08:59:42.208741 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:42.208911 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6384,"prepare.queue_time_us":39865,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":2671,"raft.queue_time_us":62377,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":10806,"spinlock_wait_cycles":9290496,"thread_start_us":576727,"threads_started":2,"wal-append.queue_time_us":527204}
I20251215 08:59:42.245110 23983 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.269366 23918 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.288501 23987 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:42.280833 24081 log.cc:927] Time spent T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20251215 08:59:42.292935 21379 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.363411 21336 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:42.275127 24074 consensus_queue.cc:1059] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
W20251215 08:59:42.363937 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 71050us
I20251215 08:59:42.301407 23762 raft_consensus.cc:697] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:42.300809 23872 raft_consensus.cc:740] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.306768 24075 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.309221 24060 consensus_queue.cc:237] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.282891 24003 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.298616 23851 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:42.313278 24038 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:42.314500 24006 consensus_meta.cc:287] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.687s user 0.007s sys 0.004s
I20251215 08:59:42.315241 23837 raft_consensus.cc:3055] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:42.205618 24064 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.319139 21262 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:42.365929 23624 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:42.336158 21264 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.392903 21260 raft_consensus.cc:1240] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:42.323650 24068 raft_consensus.cc:3055] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:42.298910 24062 raft_consensus.cc:3055] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
W20251215 08:59:42.372767 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.332163 23900 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.322546 23917 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:42.320996 24089 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:42.373692 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:42.331446 23861 consensus_queue.cc:1175] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 4 } }
I20251215 08:59:42.373592 21977 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.387252 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:42.400269 21257 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:42.401744 24088 log.cc:927] Time spent T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20251215 08:59:42.403452 24043 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.388506 24006 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.328876 21263 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:42.406177 24006 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.390064 21381 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:42.406068 24073 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:42.409607 23917 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.410367 21332 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:42.413666 24007 raft_consensus.cc:2804] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:42.414242 24007 raft_consensus.cc:697] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.415360 24007 consensus_queue.cc:237] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:42.372375 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 133190us
W20251215 08:59:42.365199 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 48751.3us
I20251215 08:59:42.413254 23872 consensus_queue.cc:260] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.416469 21257 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
W20251215 08:59:42.418848 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.387334 23837 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:42.422837 21978 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.456384 21379 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.365885 23989 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:42.369772 23830 raft_consensus.cc:3055] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:42.426142 21263 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20251215 08:59:42.458854 21263 pending_rounds.cc:85] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 6
I20251215 08:59:42.395501 24068 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.395965 24062 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:42.462553 24068 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.462801 24062 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.459198 21263 pending_rounds.cc:107] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20251215 08:59:42.458925 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:42.464133 21263 raft_consensus.cc:2889] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:42.465809 24062 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.466157 24074 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:42.463682 24068 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.453810 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:42.473562 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 83632.6us
I20251215 08:59:42.439200 23700 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:42.480070 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:42.482152 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.460971 23830 raft_consensus.cc:740] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:42.470034 24064 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:42.471683 23901 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.486182 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:42.461804 23897 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
W20251215 08:59:42.491573 24092 log.cc:927] Time spent T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20251215 08:59:42.472298 23959 consensus_queue.cc:1048] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
W20251215 08:59:42.497164 24083 log.cc:927] Time spent T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20251215 08:59:42.484266 24096 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:42.498121 23897 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:42.498798 23959 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:42.499065 23897 raft_consensus.cc:3055] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:42.499439 23959 raft_consensus.cc:697] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.504581 23846 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:42.471405 24059 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20251215 08:59:42.506319 24075 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:42.507222 24070 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.460453 23762 consensus_queue.cc:237] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.511494 23918 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:42.481367 23989 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
W20251215 08:59:42.512192 24094 log.cc:927] Time spent T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20251215 08:59:42.485209 24072 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:42.499528 23897 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.512624 23830 consensus_queue.cc:260] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.579308 23830 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:42.579149 23897 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.580044 23879 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.580717 23975 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:42.515321 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:42.582019 21257 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.483309 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.487569 24003 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.586879 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 1998) took 1177 ms. Trace:
I20251215 08:59:42.488332 24080 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:42.560393 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.563174 23989 raft_consensus.cc:697] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.571301 21977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 674) took 1059 ms. Trace:
I20251215 08:59:42.500752 23959 consensus_queue.cc:237] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.488325 24061 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:42.584367 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 101133us
I20251215 08:59:42.580641 23897 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.587307 21260 rpcz_store.cc:276] 1215 08:59:41.409119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:41.410920 (+ 1801us) service_pool.cc:225] Handling call
1215 08:59:41.412776 (+ 1856us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:42.393371 (+980595us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:42.586668 (+193297us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54002427d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:42.586830 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2253622144}
I20251215 08:59:42.589557 21977 rpcz_store.cc:276] 1215 08:59:41.511495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:41.512184 (+ 689us) service_pool.cc:225] Handling call
1215 08:59:41.512443 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:42.399775 (+887332us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:42.570994 (+171219us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b5400570154. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:42.571228 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1979973504}
I20251215 08:59:42.603209 21972 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:42.533751 23983 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:42.590322 21944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:42.603996 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 147664us
I20251215 08:59:42.567461 23917 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:42.585052 23978 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:42.589771 23918 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251215 08:59:42.632141 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.645371 21307 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:42.645910 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.647037 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.620685 21257 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:42.652113 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.653745 24042 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:42.656462 24006 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:42.657029 24006 raft_consensus.cc:697] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:42.657905 24006 consensus_queue.cc:237] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.648663 23989 consensus_queue.cc:237] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.664166 24061 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.621697 23917 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.667106 23917 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.654711 23700 consensus_queue.cc:1059] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20251215 08:59:42.667997 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:42.668706 21972 raft_consensus.cc:2368] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
I20251215 08:59:42.669929 21304 leader_election.cc:400] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Vote denied by peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831) with higher term. Message: Invalid argument: T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
I20251215 08:59:42.670212 21304 leader_election.cc:403] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251215 08:59:42.622079 24073 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:42.676941 23917 raft_consensus.cc:3060] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.686514 23700 raft_consensus.cc:3055] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:42.686987 23700 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.687804 23700 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.590598 24089 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.689405 23700 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:42.691108 24089 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.702492 21307 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.622960 23918 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:42.705765 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.712567 24070 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:42.632143 24095 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20251215 08:59:42.724884 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:42.725514 21337 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.717679 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.718582 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.720077 21232 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.720224 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.721052 21946 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:42.728111 21339 raft_consensus.cc:1240] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.729270 23978 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:42.729562 23932 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:42.721535 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:42.730180 23932 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:42.726464 21334 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:42.730737 23932 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.725430 23983 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251215 08:59:42.733453 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.734576 21335 raft_consensus.cc:2410] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20251215 08:59:42.735539 21337 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:42.735321 24075 consensus_queue.cc:1059] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251215 08:59:42.706058 21332 raft_consensus.cc:2955] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER beefe15185ac42ef9a20251007f39a56 (127.19.163.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:42.592687 24080 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.736400 23983 raft_consensus.cc:3055] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:42.628624 24059 consensus_queue.cc:1059] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.109s
W20251215 08:59:42.745260 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 128360us
I20251215 08:59:42.723522 23918 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:42.773746 21946 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:42.779073 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.782614 24073 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:42.783157 24073 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.780395 23897 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:42.785532 23897 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.787462 23897 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.771108 24100 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:42.789456 24100 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.791332 24100 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.791826 24037 raft_consensus.cc:2804] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:42.792402 24037 raft_consensus.cc:697] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:42.737334 24103 consensus_queue.cc:1059] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20251215 08:59:42.793298 24037 consensus_queue.cc:237] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.650717 24096 raft_consensus.cc:2764] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251215 08:59:42.745523 24061 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:42.793764 24103 raft_consensus.cc:3055] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:42.799345 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.799656 21409 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251215 08:59:42.804608 21410 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.800202 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.805543 21412 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.762283 23983 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:42.800868 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.808465 21407 raft_consensus.cc:2376] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7. Re-sending same reply.
I20251215 08:59:42.801193 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.809684 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:42.809782 21413 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.810428 21411 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:42.747622 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.811854 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.811798 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:42.813185 21407 raft_consensus.cc:2410] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251215 08:59:42.806704 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:42.813925 21978 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:42.815227 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.815891 21415 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.811162 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:42.820392 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 72891.4us
I20251215 08:59:42.821240 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.822395 21307 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.825122 21410 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:42.826284 21410 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20251215 08:59:42.827601 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:42.820848 21411 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:42.829856 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.830852 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.831028 21972 raft_consensus.cc:1240] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:42.829742 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.832888 21378 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:42.827112 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:42.838140 21410 raft_consensus.cc:2393] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:42.839654 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:42.806015 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 326028us
I20251215 08:59:42.850454 21337 raft_consensus.cc:1240] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:42.852254 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.852442 21337 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:42.845160 24073 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:42.834915 24096 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:42.858507 23872 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.806219 24103 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:42.845726 24072 consensus_queue.cc:1059] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.018s
I20251215 08:59:42.861423 24072 raft_consensus.cc:3055] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:42.861791 24072 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:42.873075 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:42.879278 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.884090 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.885782 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.886729 21339 raft_consensus.cc:2393] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:42.887602 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:42.888259 21339 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:42.862607 24072 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:42.892267 23976 consensus_meta.cc:287] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.529s user 0.012s sys 0.000s
I20251215 08:59:42.893903 21232 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:42.895192 21257 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251215 08:59:42.895273 21232 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:42.906919 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.908646 21232 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:42.827591 24061 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:42.910917 21946 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:42.912971 21415 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:42.915745 21262 raft_consensus.cc:1240] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:42.892676 23976 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.926512 23983 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.927292 23900 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:42.927819 23900 raft_consensus.cc:697] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:42.927973 23983 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.921926 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.922359 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.930706 21334 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:42.931785 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:42.932698 21334 raft_consensus.cc:2393] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:42.934074 21408 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20251215 08:59:42.922639 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.841722 24060 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:42.843389 23988 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:42.930531 21333 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:42.845659 24095 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251215 08:59:42.921093 24009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.028s sys 0.077s
I20251215 08:59:42.937577 21415 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20251215 08:59:42.920866 21410 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:42.939288 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:42.942109 21413 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:42.943878 21339 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:42.957581 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:42.958031 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:42.892691 24072 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:42.946637 21379 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:42.952669 21412 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:42.959323 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:42.979130 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:42.997157 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:43.001045 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:42.962605 24009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.940s user 0.028s sys 0.077s
I20251215 08:59:42.927856 23976 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:43.005335 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:42.948664 23988 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:42.954089 24060 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.008107 21413 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:42.936551 24041 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:42.909673 23984 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:42.929994 24069 consensus_queue.cc:1059] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
I20251215 08:59:42.929899 23830 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20251215 08:59:43.014955 23932 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.016454 23917 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831) with higher term. Message: Invalid argument: T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
W20251215 08:59:43.020717 24068 consensus_meta.cc:287] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.540s user 0.008s sys 0.002s
I20251215 08:59:42.909765 24075 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20251215 08:59:42.954663 23974 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:43.098830 23974 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:42.931274 23901 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251215 08:59:43.023666 24064 consensus_meta.cc:287] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.535s user 0.010s sys 0.003s
I20251215 08:59:42.928946 23900 consensus_queue.cc:237] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:42.909814 23978 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:43.105394 21412 raft_consensus.cc:1275] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251215 08:59:43.024662 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 61731.4us
I20251215 08:59:43.032804 21333 raft_consensus.cc:2393] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:43.041925 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 95355.7us
W20251215 08:59:43.037957 23624 consensus_meta.cc:287] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.644s user 0.007s sys 0.006s
I20251215 08:59:43.107343 21379 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:43.041947 21335 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:43.048786 21413 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:43.046775 23988 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:43.050755 24080 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:42.924326 24039 consensus_queue.cc:1059] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20251215 08:59:43.097519 23932 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:43.101228 23901 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:43.051064 24098 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.442s user 0.002s sys 0.000s
I20251215 08:59:43.109855 21339 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:43.123206 21379 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:43.113701 21411 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:43.047184 23975 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:43.100929 23974 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:43.051177 24060 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:43.155622 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:43.152107 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.157152 21333 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:43.163005 21379 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:43.122121 23901 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:43.170424 21232 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:43.174664 24105 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:43.175222 24105 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:43.175570 24105 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:43.177197 24064 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:43.179169 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:43.179236 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.180012 21407 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:43.129068 23976 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:43.129372 24080 raft_consensus.cc:3055] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:43.127234 24039 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:43.129123 23989 consensus_queue.cc:1059] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20251215 08:59:43.186986 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.204201 21340 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:43.209275 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.189719 24080 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:43.211987 23984 consensus_queue.cc:1059] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:43.212371 24080 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.189464 24060 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:43.213056 23984 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:43.213454 24060 raft_consensus.cc:697] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:43.213553 23984 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.214370 24060 consensus_queue.cc:237] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.215514 23984 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:43.141239 23918 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.212846 24003 raft_consensus.cc:3055] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:43.186292 24071 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:43.198154 21409 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
W20251215 08:59:43.203622 23959 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.018s sys 0.017s
I20251215 08:59:43.163631 24069 consensus_queue.cc:1059] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20251215 08:59:43.189056 24041 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20251215 08:59:43.229652 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.237358 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:43.238756 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.240648 21409 raft_consensus.cc:2393] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:43.243757 21333 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:43.244966 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.245689 21333 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:43.188714 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.247378 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.247601 24069 raft_consensus.cc:3055] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:43.226433 21160 catalog_manager.cc:5654] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 5 to 6, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 6 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:43.247934 24069 raft_consensus.cc:740] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:43.248791 24069 consensus_queue.cc:260] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.248728 21160 catalog_manager.cc:5654] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:43.250155 24069 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:43.251119 21160 catalog_manager.cc:5654] T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 1 to 4, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 4 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:43.235857 24071 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.236016 24080 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:43.269114 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:43.267650 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.202544 24039 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:43.269462 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 357965us
W20251215 08:59:43.239507 23959 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.018s sys 0.017s
I20251215 08:59:43.224057 24003 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:43.298928 24068 raft_consensus.cc:3055] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:43.299582 24068 raft_consensus.cc:740] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:43.299011 24003 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.300768 24003 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:43.269042 21307 connection.cc:232] RPC call timeout handler was delayed by -2.3935020379999514s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20251215 08:59:43.306484 21307 connection.cc:232] RPC call timeout handler was delayed by -1.3737416279999479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
W20251215 08:59:43.306782 21307 connection.cc:232] RPC call timeout handler was delayed by -1.1104150769999706s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20251215 08:59:43.307300 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:43.278333 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:43.279652 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:43.216014 23984 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20251215 08:59:43.316250 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.317587 21335 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:43.318238 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.300427 24068 consensus_queue.cc:260] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.295691 21162 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36424 (request call id 226) took 9851 ms. Trace:
I20251215 08:59:43.297205 21160 catalog_manager.cc:5654] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 3 to 7, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67). New cstate: current_term: 7 leader_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:43.294548 21414 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:43.319145 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.331740 21335 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:43.333608 21307 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:43.335059 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 312151us
I20251215 08:59:43.337769 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.338546 21332 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:43.288028 21340 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:43.282497 23911 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:43.344376 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.347018 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:43.355721 21333 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:43.357836 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.362025 21333 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:43.363044 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:43.403209 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:43.426713 21946 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:43.363922 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:43.433827 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.440640 21340 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:43.329504 21162 rpcz_store.cc:276] 1215 08:59:33.444396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:33.451884 (+ 7488us) service_pool.cc:225] Handling call
1215 08:59:43.295571 (+9843687us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:36.342824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:36.371398 (+ 28574us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:36.371447 (+ 49us) write_op.cc:435] Acquired schema lock
1215 08:59:36.371465 (+ 18us) tablet.cc:660] Decoding operations
1215 08:59:36.373314 (+ 1849us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:36.373419 (+ 105us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:36.373462 (+ 43us) tablet.cc:683] Acquiring locks for 24 operations
1215 08:59:36.437599 (+ 64137us) tablet.cc:699] Row locks acquired
1215 08:59:36.443730 (+ 6131us) write_op.cc:260] PREPARE: finished
1215 08:59:36.487762 (+ 44032us) write_op.cc:270] Start()
1215 08:59:36.551053 (+ 63291us) write_op.cc:276] Timestamp: P: 1765789176466064 usec, L: 0
1215 08:59:36.555280 (+ 4227us) op_driver.cc:348] REPLICATION: starting
1215 08:59:36.641209 (+ 85929us) log.cc:844] Serialized 9432 byte log entry
1215 08:59:36.915105 (+273896us) op_driver.cc:464] REPLICATION: finished
1215 08:59:37.780003 (+864898us) write_op.cc:301] APPLY: starting
1215 08:59:37.804228 (+ 24225us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:37.869588 (+ 65360us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:37.869711 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:38.121697 (+251986us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:38.350458 (+228761us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=24
1215 08:59:38.380930 (+ 30472us) write_op.cc:312] APPLY: finished
1215 08:59:38.697636 (+316706us) log.cc:844] Serialized 164 byte log entry
1215 08:59:38.960832 (+263196us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:38.961052 (+ 220us) write_op.cc:454] Released schema lock
1215 08:59:38.961110 (+ 58us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:38.985806 (+ 24696us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:38.985850 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":159950,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":864619,"apply.run_cpu_time_us":15255,"apply.run_wall_time_us":1205798,"num_ops":24,"prepare.queue_time_us":111678,"prepare.run_cpu_time_us":13699,"prepare.run_wall_time_us":360263,"raft.queue_time_us":133076,"raft.run_cpu_time_us":79944,"raft.run_wall_time_us":210025,"replication_time_us":359680,"thread_start_us":609311,"threads_started":5,"wal-append.queue_time_us":307037,"wal-append.run_cpu_time_us":8088,"wal-append.run_wall_time_us":2177739}]]}
I20251215 08:59:43.328835 24070 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:43.452502 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:43.453262 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.455360 21972 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:43.459827 21335 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:43.463523 21413 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:43.464785 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.467585 23900 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251215 08:59:43.468385 23900 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:43.468948 23900 raft_consensus.cc:697] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:43.470085 23900 consensus_queue.cc:237] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.349282 24061 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:43.431512 24089 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:43.434628 24062 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:43.480765 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.492877 21335 raft_consensus.cc:2376] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8. Re-sending same reply.
I20251215 08:59:43.493801 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.484328 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.496057 21333 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:43.495214 21379 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:43.507117 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.472707 21332 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:43.508335 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.509001 21335 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:43.508769 21379 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:43.509994 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.511075 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.511179 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.511683 21335 raft_consensus.cc:2376] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8. Re-sending same reply.
I20251215 08:59:43.512907 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.515584 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.477880 24041 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:43.516515 21413 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 8: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
W20251215 08:59:43.517382 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 335215us
I20251215 08:59:43.518253 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.510172 21340 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:43.517740 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.480284 24068 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:43.524410 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:43.524612 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.524996 21333 raft_consensus.cc:2376] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9. Re-sending same reply.
I20251215 08:59:43.526270 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.484004 24062 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:43.526645 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.528330 21340 raft_consensus.cc:1240] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20251215 08:59:43.515087 24064 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:43.570209 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.571090 21332 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [4.3-6.5]
I20251215 08:59:43.516492 23851 raft_consensus.cc:2804] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:43.571583 21340 raft_consensus.cc:3055] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:43.574563 21340 raft_consensus.cc:740] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:43.576135 21257 raft_consensus.cc:1240] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:43.491183 23901 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.582734 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.583333 21260 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:43.584203 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.583323 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.577724 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.483829 24089 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.577335 21332 raft_consensus.cc:1275] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:43.521732 24071 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:43.584045 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.603365 21257 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:43.578758 21340 consensus_queue.cc:260] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.604269 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.529963 24062 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:43.589906 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.605511 21340 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:43.606244 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 369004us
I20251215 08:59:43.606604 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.570574 23978 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:43.612396 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.620808 21264 raft_consensus.cc:2393] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:43.611603 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.613744 21257 raft_consensus.cc:2368] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
I20251215 08:59:43.622157 21332 raft_consensus.cc:1240] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:43.622236 21260 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:43.623668 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.624828 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.625352 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:43.625543 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.626564 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.627251 21261 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:43.627437 21338 raft_consensus.cc:1240] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 4. Current term is 5. Ops: []
W20251215 08:59:43.611429 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 184799us
I20251215 08:59:43.629541 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:43.630468 21257 raft_consensus.cc:2393] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:43.638280 21946 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:43.580560 23918 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:43.639336 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:43.640111 21259 raft_consensus.cc:2393] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:43.642021 21972 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:43.646457 21307 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:59:43.647265 21151 catalog_manager.cc:5177] ChangeConfig:REMOVE_PEER RPC for tablet 7edcdb5749ae4e59ab17648fe97390e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 5
I20251215 08:59:43.649329 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:43.650413 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:43.651248 21266 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251215 08:59:43.651528 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:43.519475 23837 consensus_queue.cc:260] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:43.637710 24115 log.cc:927] Time spent T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251215 08:59:43.602242 24006 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:43.671430 21340 raft_consensus.cc:1275] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251215 08:59:43.695968 21338 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:43.699862 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:43.711265 21340 pending_rounds.cc:85] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:43.704630 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:43.711539 21340 pending_rounds.cc:107] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251215 08:59:43.489653 21160 catalog_manager.cc:5654] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:43.711851 21340 raft_consensus.cc:2889] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:43.643112 24075 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:43.718526 21265 raft_consensus.cc:2393] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:43.737982 24061 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:43.653479 24130 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:43.718678 21340 raft_consensus.cc:1240] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:43.739960 24130 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.742280 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:43.742367 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:43.744758 21340 raft_consensus.cc:3055] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:43.745172 21340 raft_consensus.cc:740] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:43.719305 24006 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.746047 21340 consensus_queue.cc:260] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.747114 21340 raft_consensus.cc:3060] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:43.748648 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:43.696337 23978 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:43.705744 23917 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:43.697036 23918 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:43.706598 24039 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:43.781276 21338 raft_consensus.cc:1240] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:43.811738 21257 raft_consensus.cc:1240] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:43.738621 24074 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:43.696552 24041 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:43.786739 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 423733us
I20251215 08:59:43.784837 21232 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:43.794710 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:43.796396 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 59329.7us
I20251215 08:59:43.804263 21340 raft_consensus.cc:1275] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251215 08:59:43.809710 21264 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:43.814749 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 199069us
I20251215 08:59:43.785470 24060 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:43.649674 24096 consensus_queue.cc:1059] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.132s
I20251215 08:59:43.747603 24006 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:43.829455 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:43.817981 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 232501us
W20251215 08:59:43.872102 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:43.830036 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 45339.6us
I20251215 08:59:43.835817 21338 raft_consensus.cc:1275] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20251215 08:59:43.848384 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:43.812002 24110 log.cc:927] Time spent T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.464s user 0.000s sys 0.001s
I20251215 08:59:43.819767 23917 raft_consensus.cc:697] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:43.819181 24075 raft_consensus.cc:697] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
W20251215 08:59:43.825836 24114 log.cc:927] Time spent T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.381s user 0.001s sys 0.001s
I20251215 08:59:43.819458 24123 consensus_queue.cc:1059] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20251215 08:59:43.808990 23978 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:43.901263 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 62817.3us
I20251215 08:59:44.105441 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:44.106156 21972 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:43.865288 24039 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:44.110924 21944 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:44.122687 21408 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.006s sys 0.006s
W20251215 08:59:44.123474 21408 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.907s user 0.006s sys 0.006s
I20251215 08:59:44.007086 21261 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:44.105660 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:44.119529 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:43.836696 24061 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.123731 23917 consensus_queue.cc:237] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.945240 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1791) took 1118 ms. Trace:
I20251215 08:59:43.864223 24074 raft_consensus.cc:697] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:43.835181 24060 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:43.937052 24121 log.cc:927] Time spent T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.321s user 0.001s sys 0.000s
W20251215 08:59:44.214154 21258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.002s sys 0.016s
W20251215 08:59:43.931840 24124 log.cc:927] Time spent T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
W20251215 08:59:44.232031 21258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.003s sys 0.017s
W20251215 08:59:43.932449 24134 log.cc:927] Time spent T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
W20251215 08:59:43.931216 24147 log.cc:927] Time spent T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251215 08:59:44.232801 21261 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251215 08:59:44.103403 24062 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:43.944523 23700 raft_consensus.cc:3055] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:44.234848 21340 raft_consensus.cc:1275] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:44.235618 24062 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:44.176307 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 70452.5us
I20251215 08:59:44.079825 24006 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:44.126010 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:44.236961 24006 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:43.994467 24130 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:43.938158 24127 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20251215 08:59:44.238979 21338 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:44.231317 21411 rpcz_store.cc:276] 1215 08:59:42.826378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.830662 (+ 4284us) service_pool.cc:225] Handling call
1215 08:59:43.945066 (+1114404us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b540006efd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:43.945183 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":620267776}
W20251215 08:59:44.147867 23851 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.000s sys 0.016s
W20251215 08:59:43.937163 24129 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
W20251215 08:59:43.941501 24126 log.cc:927] Time spent T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.364s user 0.000s sys 0.001s
I20251215 08:59:44.473006 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1839) took 1748 ms. Trace:
I20251215 08:59:44.223783 24125 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:44.494376 24125 raft_consensus.cc:697] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:44.495626 23837 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:44.495515 24125 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:44.243392 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 137794us
W20251215 08:59:44.454594 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:44.497283 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:44.498008 23762 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:44.498136 21230 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:44.498533 23762 raft_consensus.cc:697] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:44.499087 24136 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:44.499460 24136 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.500051 24136 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:44.501106 23918 raft_consensus.cc:3055] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:44.501475 23918 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:44.502182 23918 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.503242 23918 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:44.412719 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 175043us
I20251215 08:59:44.506467 23978 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.506676 21944 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:44.455655 21260 consensus_meta.cc:287] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.829s user 0.017s sys 0.001s
I20251215 08:59:44.510154 21260 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:44.371286 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1847) took 1202 ms. Trace:
I20251215 08:59:44.510788 21332 rpcz_store.cc:276] 1215 08:59:43.168878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.634512 (+465634us) service_pool.cc:225] Handling call
1215 08:59:43.634746 (+ 234us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:43.746626 (+111880us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:43.746674 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:43.746734 (+ 60us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:43.746786 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.370954 (+624168us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54002111d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:44.371018 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.371197 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":256709504}
W20251215 08:59:44.240547 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 121119us
I20251215 08:59:44.241802 24039 raft_consensus.cc:697] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:44.463768 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 324) took 1096 ms. Trace:
I20251215 08:59:44.513602 21338 rpcz_store.cc:276] 1215 08:59:43.366353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.175540 (+809187us) service_pool.cc:225] Handling call
1215 08:59:44.238803 (+ 63263us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:44.462893 (+224090us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.463139 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:44.373519 24080 consensus_meta.cc:287] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.040s user 0.010s sys 0.000s
I20251215 08:59:44.455862 24071 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:43.937907 24089 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.174113 24141 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:44.470237 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:44.108023 24069 consensus_meta.cc:287] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.857s user 0.006s sys 0.005s
I20251215 08:59:44.474552 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 287) took 1265 ms. Trace:
I20251215 08:59:44.473454 21336 rpcz_store.cc:276] 1215 08:59:42.724806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.726815 (+ 2009us) service_pool.cc:225] Handling call
1215 08:59:42.727170 (+ 355us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:42.759130 (+ 31960us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:42.759178 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:42.770569 (+ 11391us) log.cc:844] Serialized 26 byte log entry
1215 08:59:43.146724 (+376155us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:43.540604 (+393880us) log.cc:844] Serialized 14 byte log entry
1215 08:59:43.583967 (+ 43363us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:43.584094 (+ 127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:44.472771 (+888677us) raft_consensus.cc:1685] finished
1215 08:59:44.472799 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.472950 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28945,"prepare.queue_time_us":151241,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":49584,"raft.run_cpu_time_us":723,"raft.run_wall_time_us":26421,"thread_start_us":700521,"threads_started":2,"wal-append.queue_time_us":492893}
I20251215 08:59:44.524982 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:44.526084 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1548) took 1033 ms. Trace:
I20251215 08:59:44.241412 24060 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:44.526438 21411 rpcz_store.cc:276] 1215 08:59:43.492309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.524643 (+1032334us) service_pool.cc:225] Handling call
1215 08:59:44.526011 (+ 1368us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:44.527662 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:44.222723 24095 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.529361 24095 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.531905 24095 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:43.933239 21160 catalog_manager.cc:5654] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:44.477805 24160 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.481173 24074 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.499329 23762 consensus_queue.cc:237] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.533660 24006 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.469252 24123 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251215 08:59:44.482731 24003 consensus_meta.cc:287] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.180s user 0.009s sys 0.001s
W20251215 08:59:44.484526 23988 consensus_meta.cc:287] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 1.376s user 0.006s sys 0.003s
W20251215 08:59:44.473743 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 236482us
I20251215 08:59:44.485230 24075 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.486630 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1788) took 1675 ms. Trace:
I20251215 08:59:44.488485 23932 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.489544 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1810) took 1604 ms. Trace:
W20251215 08:59:44.491430 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:44.563663 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:44.512161 23989 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.492632 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:44.493148 21977 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:44.243341 24061 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:43.937557 24128 log.cc:927] Time spent T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.235s user 0.000s sys 0.000s
I20251215 08:59:44.499439 21230 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:44.579936 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 80568.7us
I20251215 08:59:44.580260 21973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 507) took 1771 ms. Trace:
W20251215 08:59:44.580844 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 58785.9us
I20251215 08:59:44.581221 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:44.121240 24105 consensus_meta.cc:287] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.945s user 0.011s sys 0.000s
I20251215 08:59:44.582113 21977 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:44.580706 21230 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:44.583166 21304 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:44.583623 21230 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:44.580706 21973 rpcz_store.cc:276] 1215 08:59:42.808698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.808850 (+ 152us) service_pool.cc:225] Handling call
1215 08:59:42.809187 (+ 337us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:42.828225 (+ 19038us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:42.828263 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:42.843238 (+ 14975us) log.cc:844] Serialized 26 byte log entry
1215 08:59:43.227320 (+384082us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:43.227357 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:43.227418 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:44.580013 (+1352595us) raft_consensus.cc:1685] finished
1215 08:59:44.580046 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.580207 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34805,"raft.queue_time_us":19182,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":11539,"thread_start_us":291618,"threads_started":1,"wal-append.queue_time_us":616818}
I20251215 08:59:43.838593 24041 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.235554 23700 raft_consensus.cc:740] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:44.512264 21307 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:44.457769 24062 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:44.517309 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1802) took 1665 ms. Trace:
I20251215 08:59:44.517977 21332 raft_consensus.cc:1275] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251215 08:59:44.224352 23901 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:44.470541 23851 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.848s user 0.001s sys 0.031s
I20251215 08:59:44.599313 23851 raft_consensus.cc:697] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:44.599591 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1795) took 1772 ms. Trace:
I20251215 08:59:44.599673 24062 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:44.511780 24072 consensus_meta.cc:287] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 1.469s user 0.011s sys 0.001s
W20251215 08:59:44.508153 24068 consensus_meta.cc:287] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.980s user 0.010s sys 0.000s
W20251215 08:59:44.519116 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 65355.5us
I20251215 08:59:44.579486 24095 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:44.512064 24037 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251215 08:59:44.514277 23900 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.524677 24136 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.628264 23900 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.629539 24136 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.512326 23955 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:44.630149 23900 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.630314 23955 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.631214 23900 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.631726 23900 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.594059 24003 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:44.632359 23955 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.632563 24003 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:44.632953 24003 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:44.562819 21333 rpcz_store.cc:276] 1215 08:59:42.885270 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.528821 (+643551us) service_pool.cc:225] Handling call
1215 08:59:43.579155 (+ 50334us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:43.584489 (+ 5334us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:43.584531 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:43.602305 (+ 17774us) log.cc:844] Serialized 52 byte log entry
1215 08:59:43.791363 (+189058us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:43.791411 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:43.791499 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:44.488298 (+696799us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b48001eda58. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398cacde21 00007f398cac4eb7 00007f398cac4d4a 00007f398cbbbf7f 00007f398cbbbc47 00007f398cbd026c 00007f398cbd0218 00007f398cbd0198 00007f398cbd0153 00007f398cbcf3fc 00007f398bb0a516
1215 08:59:44.489246 (+ 948us) raft_consensus.cc:1685] finished
1215 08:59:44.489270 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.489484 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4900,"raft.queue_time_us":5415,"raft.run_cpu_time_us":1548,"raft.run_wall_time_us":901598,"spinlock_wait_cycles":35769728,"thread_start_us":144901,"threads_started":1,"wal-append.queue_time_us":860850}
I20251215 08:59:44.634639 21333 raft_consensus.cc:1240] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:44.634765 24068 consensus_queue.cc:1048] T 6053bc9278744c6b93f60f788aa7b06e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:44.561020 21415 rpcz_store.cc:276] 1215 08:59:42.811546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.000265 (+188719us) service_pool.cc:225] Handling call
1215 08:59:43.007638 (+ 7373us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:43.023527 (+ 15889us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:43.023575 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:43.024135 (+ 560us) log.cc:844] Serialized 26 byte log entry
1215 08:59:43.271198 (+247063us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:43.271238 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:43.271320 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:44.486419 (+1215099us) raft_consensus.cc:1685] finished
1215 08:59:44.486442 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.486569 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66599,"raft.queue_time_us":24354,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":6160,"thread_start_us":245402,"threads_started":1,"wal-append.queue_time_us":478737}
I20251215 08:59:44.635948 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:44.638324 21260 raft_consensus.cc:1240] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:44.596181 21341 rpcz_store.cc:276] 1215 08:59:42.852150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.853242 (+ 1092us) service_pool.cc:225] Handling call
1215 08:59:42.853518 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:42.936454 (+ 82936us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:42.936482 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:42.936523 (+ 41us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:42.936549 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.516990 (+1580441us) spinlock_profiling.cc:243] Waited 82.7 ms on lock 0x7b5400047c54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:44.517133 (+ 143us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.517253 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":190306688}
I20251215 08:59:44.596890 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1854) took 1143 ms. Trace:
I20251215 08:59:44.641422 21332 rpcz_store.cc:276] 1215 08:59:43.454000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.516597 (+1062597us) service_pool.cc:225] Handling call
1215 08:59:44.517472 (+ 875us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:44.596661 (+ 79189us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.596842 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:44.567811 23988 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.600065 21412 rpcz_store.cc:276] 1215 08:59:42.827298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.040963 (+213665us) service_pool.cc:225] Handling call
1215 08:59:43.041407 (+ 444us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:43.105773 (+ 64366us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.599350 (+1493577us) spinlock_profiling.cc:243] Waited 63.6 ms on lock 0x7b54000e9f54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:44.599534 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146239744}
I20251215 08:59:44.657640 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:44.525344 21408 rpcz_store.cc:276] 1215 08:59:43.209496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.215222 (+ 5726us) service_pool.cc:225] Handling call
1215 08:59:43.215511 (+ 289us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:43.215712 (+ 201us) raft_consensus.cc:1534] Early marking committed up to index 9
1215 08:59:44.222013 (+1006301us) log.cc:844] Serialized 14 byte log entry
1215 08:59:44.474215 (+252202us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:44.474274 (+ 59us) raft_consensus.cc:1645] Marking committed up to 9
1215 08:59:44.474319 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.474384 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.474503 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103322,"prepare.queue_time_us":1267317,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":103597,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":1107944,"threads_started":2,"wal-append.queue_time_us":269326}
I20251215 08:59:44.658474 21412 raft_consensus.cc:2393] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:44.548203 24071 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.605094 24064 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.677874 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:44.678665 21972 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:44.549474 24141 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.591409 24105 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.621218 23978 consensus_queue.cc:1059] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20251215 08:59:44.532788 24160 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.632961 24069 consensus_queue.cc:1059] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20251215 08:59:44.593881 24123 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:44.582800 23932 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.583149 23989 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.622855 24037 consensus_queue.cc:1048] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251215 08:59:44.593660 24080 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:44.591867 24006 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:44.579761 24089 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.645368 24095 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.676330 24068 consensus_queue.cc:1059] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20251215 08:59:44.692517 24104 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.677871 24064 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.692386 24140 consensus_queue.cc:1059] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.226s
W20251215 08:59:44.740101 24133 log.cc:927] Time spent T dd5a10a353d64f5c826c27e4a01f6132 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.265s user 0.000s sys 0.001s
W20251215 08:59:44.746217 24156 log.cc:927] Time spent T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
W20251215 08:59:44.755018 24154 log.cc:927] Time spent T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20251215 08:59:44.756999 24141 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.758358 24160 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.761871 24105 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:44.768484 24175 raft_consensus.cc:3055] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:44.769091 24175 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:44.770220 24175 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.905943 23837 consensus_queue.cc:1048] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:44.908808 24175 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:44.601272 21263 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:44.800778 21266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2049) took 1148 ms. Trace:
W20251215 08:59:44.807953 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 295530us
I20251215 08:59:44.646337 21332 raft_consensus.cc:1275] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251215 08:59:44.920866 24141 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.921496 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:44.922547 21332 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:44.609899 21335 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20251215 08:59:44.819379 21304 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:44.609505 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1805) took 1725 ms. Trace:
W20251215 08:59:44.924103 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 111672us
I20251215 08:59:44.923928 21339 rpcz_store.cc:276] 1215 08:59:42.883832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.957712 (+ 73880us) service_pool.cc:225] Handling call
1215 08:59:44.609272 (+1651560us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540009bfd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:44.609409 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":347286400}
I20251215 08:59:44.924906 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:44.925213 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 311272us
I20251215 08:59:44.822846 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1806) took 1938 ms. Trace:
I20251215 08:59:44.835942 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 283) took 2036 ms. Trace:
I20251215 08:59:44.601171 21407 raft_consensus.cc:1240] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:44.658653 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:44.632433 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:44.658953 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1543) took 1316 ms. Trace:
I20251215 08:59:44.905508 24003 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.930174 21412 rpcz_store.cc:276] 1215 08:59:43.342480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.657379 (+1314899us) service_pool.cc:225] Handling call
1215 08:59:44.658902 (+ 1523us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:44.930431 24072 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:44.931175 24072 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:44.931689 24072 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:44.932179 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:44.651101 21415 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:44.933401 24177 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.933689 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1549) took 1430 ms. Trace:
I20251215 08:59:44.837577 21409 raft_consensus.cc:2368] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8.
I20251215 08:59:44.838617 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 325) took 1008 ms. Trace:
W20251215 08:59:44.659966 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 146151us
I20251215 08:59:44.862489 24182 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:44.934742 21336 rpcz_store.cc:276] 1215 08:59:43.830512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.525576 (+695064us) service_pool.cc:225] Handling call
1215 08:59:44.525846 (+ 270us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:44.526035 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 9
1215 08:59:44.526289 (+ 254us) log.cc:844] Serialized 14 byte log entry
1215 08:59:44.837736 (+311447us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:44.837848 (+ 112us) raft_consensus.cc:1645] Marking committed up to 9
1215 08:59:44.837988 (+ 140us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.838126 (+ 138us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.838382 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":121,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":42442,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":135,"spinlock_wait_cycles":8064,"thread_start_us":310736,"threads_started":1,"wal-append.queue_time_us":187426}
I20251215 08:59:44.627240 23700 consensus_queue.cc:260] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.936926 21378 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:44.938824 23700 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:44.600188 23851 consensus_queue.cc:237] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:44.940670 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:44.941403 24178 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:44.934234 21415 rpcz_store.cc:276] 1215 08:59:43.503303 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.635725 (+1132422us) service_pool.cc:225] Handling call
1215 08:59:44.933467 (+297742us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:44.871376 23900 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.697417 23988 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.942536 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:44.876997 23918 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.943229 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:44.943770 23918 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.941831 24178 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:44.877418 24004 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.878679 24061 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.879549 23932 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.947633 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:44.933565 24164 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:44.948596 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:44.949164 24164 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.949688 24164 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:44.712870 24089 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.945679 24178 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:44.953699 24089 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.676312 24039 consensus_queue.cc:237] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.882130 24085 consensus_queue.cc:1059] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.369s
I20251215 08:59:44.958137 24085 raft_consensus.cc:3055] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:44.958448 24085 raft_consensus.cc:740] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:44.959532 24085 consensus_queue.cc:260] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.960143 21415 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:44.960736 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1554) took 1326 ms. Trace:
I20251215 08:59:44.961098 21415 rpcz_store.cc:276] 1215 08:59:43.634619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.942791 (+1308172us) service_pool.cc:225] Handling call
1215 08:59:44.960659 (+ 17868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:44.888111 21257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1534) took 1248 ms. Trace:
I20251215 08:59:44.702126 24006 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:44.901257 24140 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.886118 24041 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.744120 24104 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.902701 24170 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.961884 21257 rpcz_store.cc:276] 1215 08:59:43.639235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.639844 (+ 609us) service_pool.cc:225] Handling call
1215 08:59:43.811550 (+171706us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:43.812176 (+ 626us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.887550 (+1075374us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b5400289950. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:44.887862 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":394156416}
I20251215 08:59:44.965340 24170 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.965785 24140 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.966619 24104 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:44.904556 24095 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:44.905485 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 263951us
I20251215 08:59:44.968317 24140 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.936709 24182 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.882629 23989 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:44.702196 24080 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:44.900631 24007 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:44.910097 21266 rpcz_store.cc:276] 1215 08:59:43.652395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.708194 (+ 55799us) service_pool.cc:225] Handling call
1215 08:59:43.748875 (+ 40681us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:43.749171 (+ 296us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:43.749195 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:43.749655 (+ 460us) log.cc:844] Serialized 26 byte log entry
1215 08:59:43.891298 (+141643us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:43.891361 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:43.891462 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:44.799088 (+907626us) raft_consensus.cc:1685] finished
1215 08:59:44.799129 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.800435 (+ 1306us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":141278,"threads_started":1,"wal-append.queue_time_us":806853}
I20251215 08:59:44.971067 24080 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:44.927565 21406 rpcz_store.cc:276] 1215 08:59:42.799237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.799787 (+ 550us) service_pool.cc:225] Handling call
1215 08:59:42.828401 (+ 28614us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:42.828583 (+ 182us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:42.828610 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:44.571736 (+1743126us) spinlock_profiling.cc:243] Waited 7.1 ms on lock 0x7b2c00092518. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:44.583616 (+ 11880us) log.cc:844] Serialized 26 byte log entry
1215 08:59:44.834373 (+250757us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:44.834673 (+ 300us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.834860 (+ 187us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:44.834987 (+ 127us) raft_consensus.cc:1685] finished
1215 08:59:44.835015 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.835554 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16341248,"thread_start_us":90123,"threads_started":1,"wal-append.queue_time_us":163670}
I20251215 08:59:44.973400 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:44.974298 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1557) took 1338 ms. Trace:
I20251215 08:59:44.974632 21406 rpcz_store.cc:276] 1215 08:59:43.635604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.972934 (+1337330us) service_pool.cc:225] Handling call
1215 08:59:44.974232 (+ 1298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:44.921875 24141 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.975777 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:44.611505 21411 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:44.928913 23901 consensus_queue.cc:1048] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
W20251215 08:59:44.977643 21411 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1544) took 1543 ms (client timeout 2041 ms). Trace:
I20251215 08:59:44.977783 23932 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:44.978174 21411 rpcz_store.cc:269] 1215 08:59:43.434147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.527358 (+1093211us) service_pool.cc:225] Handling call
1215 08:59:44.977445 (+450087us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:44.978777 21414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1781) took 2168 ms (client timeout 1972 ms). Trace:
I20251215 08:59:44.978641 23932 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:44.979223 21414 rpcz_store.cc:269] 1215 08:59:42.809752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.811542 (+ 1790us) service_pool.cc:225] Handling call
1215 08:59:44.978366 (+2166824us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b5400341f54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:44.978559 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":721187840}
I20251215 08:59:44.971473 21975 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 1.
W20251215 08:59:44.918570 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 226313us
I20251215 08:59:44.934237 24177 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.926784 21334 rpcz_store.cc:276] 1215 08:59:42.884122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.041771 (+157649us) service_pool.cc:225] Handling call
1215 08:59:43.042164 (+ 393us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:43.042669 (+ 505us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:43.042711 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:43.047011 (+ 4300us) log.cc:844] Serialized 52 byte log entry
1215 08:59:43.342814 (+295803us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:43.437785 (+ 94971us) log.cc:844] Serialized 14 byte log entry
1215 08:59:44.071867 (+634082us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x7b1800349660. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b03045 00007f3987babce0 00007f3987bbe396 00007f3987bcef07 00007f3987bcee88
1215 08:59:44.148134 (+ 76267us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.150363 (+ 2229us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:44.822092 (+671729us) raft_consensus.cc:1685] finished
1215 08:59:44.822294 (+ 202us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.822625 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":92002,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":4977,"raft.queue_time_us":21273,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":17926,"spinlock_wait_cycles":93912832,"thread_start_us":307058,"threads_started":2,"wal-append.queue_time_us":502290}
I20251215 08:59:44.878167 23955 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.946229 21412 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:44.947278 23918 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.946857 24004 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.948498 21979 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:44.674085 23974 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.969372 24170 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.970253 23978 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:44.975818 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 322) took 1047 ms. Trace:
I20251215 08:59:44.979565 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:44.980597 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:44.914628 24064 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:44.927371 21307 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:44.976732 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1546) took 1519 ms. Trace:
I20251215 08:59:44.932782 24003 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:44.965361 24006 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:44.987939 23974 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:44.984830 24041 consensus_queue.cc:1048] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20251215 08:59:44.987964 23955 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.991434 21406 rpcz_store.cc:276] 1215 08:59:43.456900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.975344 (+1518444us) service_pool.cc:225] Handling call
1215 08:59:44.976677 (+ 1333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:44.993245 24177 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:44.993427 24105 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:44.994206 23978 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:44.994516 23872 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:45.000751 23911 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:45.000883 24003 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:45.026993 24003 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:45.001612 23974 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.032249 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.033737 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.034440 24182 leader_election.cc:290] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.035193 21258 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
W20251215 08:59:45.037201 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:45.038439 24068 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:45.039652 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.040187 24068 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:44.971153 24007 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.041842 21408 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:45.042654 24103 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:44.988432 24141 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.007445 23955 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.045682 21381 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:45.046229 24105 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.049226 24100 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.024163 24075 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:45.049957 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:45.050366 23976 raft_consensus.cc:2804] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:45.021090 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.012182 21414 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:45.022102 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:44.995445 24085 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:45.040441 21266 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:44.990228 21259 rpcz_store.cc:276] 1215 08:59:43.928458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.928697 (+ 239us) service_pool.cc:225] Handling call
1215 08:59:44.974484 (+1045787us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:44.974763 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:44.975224 (+ 461us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:44.975263 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:44.975357 (+ 94us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:44.975452 (+ 95us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.975521 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:44.975630 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":115,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":244,"spinlock_wait_cycles":22981888}
I20251215 08:59:44.999487 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.050616 21260 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:45.051630 21411 raft_consensus.cc:2393] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:45.058955 21332 raft_consensus.cc:2468] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:45.059903 21412 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:45.062320 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.062608 23955 raft_consensus.cc:2764] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251215 08:59:45.064471 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.065441 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1547) took 1587 ms. Trace:
I20251215 08:59:45.065886 21406 rpcz_store.cc:276] 1215 08:59:43.478524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.064043 (+1585519us) service_pool.cc:225] Handling call
1215 08:59:45.065370 (+ 1327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.061116 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.050253 24075 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:45.087278 21409 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1537) took 1833 ms (client timeout 1683 ms). Trace:
I20251215 08:59:45.066547 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:45.464027 21409 rpcz_store.cc:269] 1215 08:59:43.253915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.277987 (+ 24072us) service_pool.cc:225] Handling call
1215 08:59:45.087042 (+1809055us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54001df6d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.087192 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3506841344}
I20251215 08:59:45.116663 24003 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.178475 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:45.465893 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:45.466205 21406 raft_consensus.cc:2376] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7. Re-sending same reply.
I20251215 08:59:45.467132 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:45.058559 23976 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:45.469843 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:45.319586 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 200087us
I20251215 08:59:45.311955 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:45.201771 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:45.285856 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:45.467876 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 540696us
I20251215 08:59:45.225430 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.349938 21408 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:45.473672 21411 raft_consensus.cc:2376] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6. Re-sending same reply.
W20251215 08:59:45.086887 21263 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1514) took 1562 ms (client timeout 1975 ms). Trace:
W20251215 08:59:45.487757 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:45.074975 21412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59102 (request call id 288) took 1725 ms (client timeout 2186 ms). Trace:
W20251215 08:59:45.489558 21412 rpcz_store.cc:269] 1215 08:59:43.336355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.931604 (+1595249us) service_pool.cc:225] Handling call
1215 08:59:45.061466 (+129862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.326165 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 326) took 1495 ms. Trace:
I20251215 08:59:45.490376 21258 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:45.055886 23911 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:45.492038 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:45.083551 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:45.049973 24100 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.510246 21415 raft_consensus.cc:2393] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:45.510959 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:45.475208 23976 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:45.513041 24100 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.271342 24178 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.515499 24178 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.516409 21260 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:45.467414 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:45.517621 21406 raft_consensus.cc:606] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:45.517715 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.518066 21406 raft_consensus.cc:614] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:45.520161 21406 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20251215 08:59:45.139679 23872 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.505342 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:45.478243 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.525938 21979 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:45.520831 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1799) took 1998 ms. Trace:
I20251215 08:59:45.528229 21406 rpcz_store.cc:276] 1215 08:59:43.522450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.519148 (+1996698us) service_pool.cc:225] Handling call
1215 08:59:45.519683 (+ 535us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:45.520525 (+ 842us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.520757 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.088477 23901 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:45.485591 21263 rpcz_store.cc:269] 1215 08:59:43.523925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.524328 (+ 403us) service_pool.cc:225] Handling call
1215 08:59:45.086435 (+1562107us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54000cc454. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.086654 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2280577792}
I20251215 08:59:45.109118 24071 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:45.462370 24188 log.cc:927] Time spent T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20251215 08:59:45.492869 24164 consensus_meta.cc:287] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.543s user 0.011s sys 0.002s
I20251215 08:59:45.322319 23700 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.129827 23955 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:45.202615 24186 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20251215 08:59:45.222889 24060 consensus_meta.cc:287] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.696s user 0.008s sys 0.016s
I20251215 08:59:45.087911 23978 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.524902 23872 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.553980 24070 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:45.554544 21406 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251215 08:59:45.558434 23976 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.445384 24004 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.592486 21261 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:45.369060 24105 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.419828 23851 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.592106 21332 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:45.490545 21338 rpcz_store.cc:276] 1215 08:59:43.830870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.551353 (+720483us) service_pool.cc:225] Handling call
1215 08:59:44.612938 (+ 61585us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:44.613336 (+ 398us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:44.613376 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:44.613434 (+ 58us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:44.613478 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:44.613570 (+ 92us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:45.325717 (+712147us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b540009bfd0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.325927 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8544,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":73421184}
I20251215 08:59:45.582926 21339 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:45.584642 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.595916 24075 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.597537 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1552) took 1964 ms. Trace:
I20251215 08:59:45.606199 21406 rpcz_store.cc:276] 1215 08:59:43.621351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.553548 (+1932197us) service_pool.cc:225] Handling call
1215 08:59:45.554060 (+ 512us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.582876 (+ 28816us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.585404 (+ 2528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.608317 24070 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:45.608457 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 97617.2us
W20251215 08:59:45.612846 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:45.569474 24164 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.621165 21307 leader_election.cc:400] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6.
I20251215 08:59:45.594688 23700 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.626118 21412 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
W20251215 08:59:45.621068 24175 consensus_meta.cc:287] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.711s user 0.008s sys 0.005s
W20251215 08:59:45.621620 24072 consensus_meta.cc:287] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.689s user 0.012s sys 0.001s
I20251215 08:59:45.608940 24070 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.632707 21334 raft_consensus.cc:1240] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:45.634032 24164 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.588534 24060 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.628554 24075 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.004s
I20251215 08:59:45.634907 24070 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.638782 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1819) took 2285 ms. Trace:
I20251215 08:59:45.639285 21340 rpcz_store.cc:276] 1215 08:59:43.352874 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.828350 (+475476us) service_pool.cc:225] Handling call
1215 08:59:43.828626 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:44.235472 (+406846us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.638358 (+1402886us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b5400298454. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:45.638613 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":933229952}
I20251215 08:59:45.639878 21264 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:45.631695 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1545) took 2183 ms. Trace:
I20251215 08:59:45.641021 21412 rpcz_store.cc:276] 1215 08:59:43.444399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.526361 (+2081962us) service_pool.cc:225] Handling call
1215 08:59:45.590655 (+ 64294us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.627275 (+ 36620us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.627495 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:45.622298 23974 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.014s sys 0.022s
I20251215 08:59:45.599505 23901 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.642207 21406 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:45.634938 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:45.642978 21409 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:45.643061 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:45.622905 23955 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.650873 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1556) took 2015 ms. Trace:
I20251215 08:59:45.651149 21406 rpcz_store.cc:276] 1215 08:59:43.635282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.618836 (+1983554us) service_pool.cc:225] Handling call
1215 08:59:45.635930 (+ 17094us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.650662 (+ 14732us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.650828 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.652321 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:45.656392 24190 log.cc:927] Time spent T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20251215 08:59:45.649498 23762 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:45.644184 23974 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.594s user 0.014s sys 0.022s
I20251215 08:59:45.657989 21257 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:45.658152 24123 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:45.641839 23978 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:45.652287 24073 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:45.631811 23851 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:45.659004 24080 consensus_meta.cc:287] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.687s user 0.005s sys 0.009s
I20251215 08:59:45.665975 21406 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b83855234bda47a1b06c3f932728766b"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:45.666317 24004 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:45.636689 24072 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.667811 21336 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 2.
I20251215 08:59:45.674116 21333 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:45.657042 23762 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.679067 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:45.669392 24073 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:45.703627 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 94379.6us
I20251215 08:59:45.681265 24004 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:45.713788 24059 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:45.714622 24059 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.717247 23978 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.719830 23978 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:45.720584 21415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1553) took 2086 ms (client timeout 2127 ms). Trace:
W20251215 08:59:45.721215 21415 rpcz_store.cc:269] 1215 08:59:43.634311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.984279 (+1349968us) service_pool.cc:225] Handling call
1215 08:59:45.720354 (+736075us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54000eae54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.720487 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1037904128}
I20251215 08:59:45.723464 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1559) took 1214 ms. Trace:
I20251215 08:59:45.723919 21415 rpcz_store.cc:276] 1215 08:59:44.508583 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.722244 (+1213661us) service_pool.cc:225] Handling call
1215 08:59:45.722696 (+ 452us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.723072 (+ 376us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:45.723102 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:45.723178 (+ 76us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:45.723217 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.723278 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:45.723401 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.732419 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1604) took 2876 ms. Trace:
I20251215 08:59:45.735842 24069 raft_consensus.cc:3055] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:45.736506 24069 raft_consensus.cc:740] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:45.709949 24123 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.739501 24123 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:45.740005 21408 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1550) took 2219 ms (client timeout 1989 ms). Trace:
I20251215 08:59:45.742985 24164 consensus_queue.cc:1059] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20251215 08:59:45.740617 21408 rpcz_store.cc:269] 1215 08:59:43.520864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.658437 (+1137573us) service_pool.cc:225] Handling call
1215 08:59:45.739743 (+1081306us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54000bcf54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.739906 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259941632}
I20251215 08:59:45.754418 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:45.755187 21979 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:45.705667 24073 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:45.733009 21337 rpcz_store.cc:276] 1215 08:59:42.855322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:42.855512 (+ 190us) service_pool.cc:225] Handling call
1215 08:59:45.730808 (+2875296us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398cb6d58e 00007f398610c437 00007f398610c3c2 00007f398610c2f1 00007f398610c26c 00007f398610b522 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.732144 (+ 1336us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3969880960}
I20251215 08:59:45.754117 21409 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:45.758819 21339 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:45.681785 24080 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.747107 24004 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:45.766180 21260 raft_consensus.cc:1240] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:45.756098 23976 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.771706 21973 raft_consensus.cc:2410] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:45.766089 24059 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:45.775915 24085 consensus_meta.cc:287] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.707s user 0.010s sys 0.003s
I20251215 08:59:45.795895 23700 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.783725 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1789) took 2936 ms. Trace:
I20251215 08:59:45.803360 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.819312 24196 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.830168 21259 raft_consensus.cc:2410] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:45.807173 24072 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.830688 24196 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.783480 23917 raft_consensus.cc:3055] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:45.831358 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.801643 24164 consensus_queue.cc:1059] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20251215 08:59:45.831660 23917 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:45.798883 21408 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251215 08:59:45.784857 23955 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.819437 24142 raft_consensus.cc:3055] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:45.833356 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 290) took 1154 ms. Trace:
I20251215 08:59:45.833467 24142 raft_consensus.cc:740] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:45.783614 24069 consensus_queue.cc:260] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.806535 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 324) took 1875 ms. Trace:
I20251215 08:59:45.804092 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:45.835322 23851 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:45.833367 23917 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.798385 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1538) took 2529 ms. Trace:
W20251215 08:59:45.811823 21265 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1521) took 2228 ms (client timeout 2150 ms). Trace:
I20251215 08:59:45.836429 23917 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:45.837047 24072 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:45.826561 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59102 (request call id 291) took 1147 ms. Trace:
I20251215 08:59:45.768815 24123 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:45.837831 24072 raft_consensus.cc:697] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:45.837949 21414 rpcz_store.cc:276] 1215 08:59:44.679269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.980188 (+300919us) service_pool.cc:225] Handling call
1215 08:59:45.826358 (+846170us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b54001761d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.826495 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49780608}
I20251215 08:59:45.838423 24123 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.838968 24072 consensus_queue.cc:237] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.839672 21414 raft_consensus.cc:1275] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20251215 08:59:45.840185 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:45.840252 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 292) took 1160 ms. Trace:
I20251215 08:59:45.807408 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.841516 24123 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.842052 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:45.836308 21407 rpcz_store.cc:276] 1215 08:59:43.268561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.269078 (+ 517us) service_pool.cc:225] Handling call
1215 08:59:44.600959 (+1331881us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:44.937389 (+336430us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.798056 (+860667us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54000e9f50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.798218 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3049653632}
I20251215 08:59:45.842796 21261 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:45.843199 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:45.844445 24100 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.844574 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.844986 24100 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.791143 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.834793 24142 consensus_queue.cc:260] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.846235 24142 raft_consensus.cc:3060] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:45.806660 23762 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.846982 24100 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.847472 24100 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.836544 21307 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:45.793944 23901 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:45.848390 21378 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:45.836773 21265 rpcz_store.cc:269] 1215 08:59:43.582992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.583108 (+ 116us) service_pool.cc:225] Handling call
1215 08:59:45.811374 (+2228266us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540020b554. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:45.811612 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305217280}
I20251215 08:59:45.850157 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.805716 24007 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.850440 24105 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.851336 24085 consensus_queue.cc:1059] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:45.802837 23932 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:45.852310 24085 raft_consensus.cc:3055] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:45.851917 24105 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.846182 21260 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:45.819888 21381 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:45.840770 21414 rpcz_store.cc:276] 1215 08:59:44.679568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.838929 (+1159361us) service_pool.cc:225] Handling call
1215 08:59:45.839330 (+ 401us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.840018 (+ 688us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.840202 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.835454 21264 rpcz_store.cc:276] 1215 08:59:43.931013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.125185 (+194172us) service_pool.cc:225] Handling call
1215 08:59:44.125507 (+ 322us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.640451 (+1514944us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.806261 (+165810us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54001342d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:45.806475 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3481837312}
I20251215 08:59:45.852797 23955 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:45.852665 24085 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:45.837404 24164 raft_consensus.cc:3055] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:45.852741 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.833794 21408 rpcz_store.cc:276] 1215 08:59:44.678971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.756633 (+1077662us) service_pool.cc:225] Handling call
1215 08:59:45.773660 (+ 17027us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.833101 (+ 59441us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.833299 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.886932 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:45.885299 21415 raft_consensus.cc:1275] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251215 08:59:45.906644 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 86820.7us
I20251215 08:59:45.912456 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.913893 21260 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
W20251215 08:59:45.771884 24006 consensus_meta.cc:287] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.771s user 0.008s sys 0.006s
I20251215 08:59:45.795989 24080 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.916345 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.912511 21415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1800) took 1225 ms. Trace:
I20251215 08:59:45.918563 21415 rpcz_store.cc:276] 1215 08:59:44.677276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.756574 (+1079298us) service_pool.cc:225] Handling call
1215 08:59:45.773329 (+ 16755us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:45.902466 (+129137us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.902742 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:45.886986 23837 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:45.897181 24164 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:45.919708 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.920099 23837 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.920671 24164 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.922044 24164 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:45.923491 23837 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.834743 24196 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.924655 21406 raft_consensus.cc:606] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:45.921097 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.834935 24069 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
W20251215 08:59:45.985889 24039 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.037s sys 0.053s
I20251215 08:59:45.930528 21265 raft_consensus.cc:2393] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 8: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:45.771588 24060 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:45.847965 24100 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.986409 21979 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:45.977676 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:45.913355 23901 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.013545 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:45.977494 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:45.803089 23911 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:45.930927 23762 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:45.988122 21381 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:45.929363 24070 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:45.829671 21410 rpcz_store.cc:276] 1215 08:59:42.825883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.002189 (+176306us) service_pool.cc:225] Handling call
1215 08:59:43.002617 (+ 428us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:43.021848 (+ 19231us) raft_consensus.cc:1534] Early marking committed up to index 2
1215 08:59:43.021911 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:43.022505 (+ 594us) log.cc:844] Serialized 26 byte log entry
1215 08:59:43.416636 (+394131us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:43.416696 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:43.416794 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:45.744042 (+2327248us) spinlock_profiling.cc:243] Waited 9.06 ms on lock 0x7b240011f828. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f398495a9a4 00007f3984958aab 00007f39849e79a9 00007f39849e7635 00007f39849e6b90 00007f398777b828 00007f398777bf88 00007f3987753738 00007f398774eb0a 00007f398774df56
1215 08:59:45.747929 (+ 3887us) raft_consensus.cc:1685] finished
1215 08:59:45.747960 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:45.762528 (+ 14568us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14074,"raft.queue_time_us":15475,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"spinlock_wait_cycles":20843392,"thread_start_us":393659,"threads_started":1,"wal-append.queue_time_us":597549}
I20251215 08:59:46.027868 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:45.925451 24006 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:45.929934 24068 consensus_queue.cc:1059] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20251215 08:59:46.003654 23897 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
W20251215 08:59:46.012775 24039 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.050s user 0.041s sys 0.057s
I20251215 08:59:46.038625 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:46.042711 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 54698.9us
I20251215 08:59:46.032172 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:46.050555 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.052886 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:46.049185 21333 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:46.029909 23851 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:46.061817 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.061673 23897 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.063462 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.064010 24080 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:46.064178 23897 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:46.058579 24007 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:46.064594 24080 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.059820 21414 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:46.069954 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.066056 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:46.072646 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.038435 24096 raft_consensus.cc:493] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.077596 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.065220 21337 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:46.066488 23851 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:46.035481 23901 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.112154 21339 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
W20251215 08:59:46.113274 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 60476.6us
W20251215 08:59:46.117750 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:46.120424 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.038478 24070 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:46.121762 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:46.122452 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.123086 24198 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:46.124758 24198 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.125424 24198 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.127774 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.128031 21339 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:46.128499 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:46.067862 23762 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.162932 21410 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251215 08:59:46.210083 21338 raft_consensus.cc:3060] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:46.082211 24007 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:46.211148 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:46.152190 21341 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:46.153581 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.212625 21408 raft_consensus.cc:1240] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:46.219852 21972 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:46.153512 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.057578 24085 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.225610 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:46.149120 24096 raft_consensus.cc:515] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.152048 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:46.127393 23897 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:46.228688 21977 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 9: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:46.119560 23851 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:46.155732 21232 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.628s (SENT) [suppressed 30 similar messages]
I20251215 08:59:46.156231 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.155128 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.152869 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:46.128293 24105 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.150492 24070 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.131026 23955 raft_consensus.cc:697] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:46.357510 21978 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:46.358278 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 646) took 1418 ms. Trace:
I20251215 08:59:46.358652 21978 rpcz_store.cc:276] 1215 08:59:44.940253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.948062 (+ 7809us) service_pool.cc:225] Handling call
1215 08:59:46.156014 (+1207952us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b58001f3898. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8dec 00007f3984ac89ab 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:46.358190 (+202176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3877504}
I20251215 08:59:46.361027 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.361263 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 320) took 2490 ms. Trace:
W20251215 08:59:46.362178 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.361739 21262 rpcz_store.cc:276] 1215 08:59:43.870973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.872811 (+ 1838us) service_pool.cc:225] Handling call
1215 08:59:43.873270 (+ 459us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:43.873668 (+ 398us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:43.873707 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:43.875932 (+ 2225us) log.cc:844] Serialized 26 byte log entry
1215 08:59:45.656266 (+1780334us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18002082a0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:46.061268 (+405002us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:46.061314 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:46.061392 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:46.360768 (+299376us) raft_consensus.cc:1685] finished
1215 08:59:46.360792 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:46.361082 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1990,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":52258,"spinlock_wait_cycles":53735680,"thread_start_us":396772,"threads_started":1,"wal-append.queue_time_us":2357017}
I20251215 08:59:46.363330 24207 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.298363 24007 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.363962 24207 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.365841 23900 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:46.366078 24007 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.156759 24009 raft_consensus.cc:3055] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:46.366359 24207 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.366844 24009 raft_consensus.cc:740] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:46.366454 23900 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.367571 24100 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:46.368247 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1857) took 1723 ms. Trace:
I20251215 08:59:46.352447 24006 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.368049 24009 consensus_queue.cc:260] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.322530 23861 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.370358 24198 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:46.370180 23861 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.370633 23762 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:46.368883 23900 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:46.372323 23911 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:46.372144 23901 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.372135 23861 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:46.335431 21332 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:46.322310 23965 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.374611 23965 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:46.375026 23917 consensus_meta.cc:287] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.538s user 0.008s sys 0.001s
W20251215 08:59:46.282506 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.238344 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:46.334936 21266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2066) took 1274 ms. Trace:
I20251215 08:59:46.376025 21257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 2060) took 1369 ms. Trace:
I20251215 08:59:46.377130 21266 rpcz_store.cc:276] 1215 08:59:45.060935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:46.334069 (+1273134us) service_pool.cc:225] Handling call
1215 08:59:46.334460 (+ 391us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:46.334639 (+ 179us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:46.334663 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:46.334717 (+ 54us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:46.334743 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:46.334789 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:46.334883 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:46.377547 21257 rpcz_store.cc:276] 1215 08:59:45.007106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.022864 (+ 15758us) service_pool.cc:225] Handling call
1215 08:59:46.375829 (+1352965us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54002e54d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:46.375939 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436833536}
I20251215 08:59:46.356981 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.368665 21341 rpcz_store.cc:276] 1215 08:59:44.644495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:44.644622 (+ 127us) service_pool.cc:225] Handling call
1215 08:59:44.644961 (+ 339us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:46.237733 (+1592772us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:46.368001 (+130268us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400115b54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:46.368169 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3274666880}
I20251215 08:59:46.380913 21974 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:46.256592 21340 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:46.155941 24196 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:46.383337 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:46.255431 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.384307 21338 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:46.222522 24073 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.295858 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:46.323948 21261 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 3.
I20251215 08:59:46.389449 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:46.389496 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:46.368912 24200 log.cc:927] Time spent T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20251215 08:59:46.391345 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.392613 21415 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 9.
I20251215 08:59:46.392938 21408 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 5: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:46.337870 24195 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.205s user 0.000s sys 0.000s
W20251215 08:59:46.687579 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.421440 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:46.386085 24004 consensus_meta.cc:287] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.610s user 0.002s sys 0.009s
I20251215 08:59:46.659641 24103 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:46.517431 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.714155 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:46.715401 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.716043 21410 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:46.697463 24006 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:46.491888 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.399353 24072 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:46.440341 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 284587us
I20251215 08:59:46.573554 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.237639 23897 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:46.421790 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:46.472115 23955 consensus_queue.cc:237] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.397289 24197 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.569995 21977 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:46.398370 21265 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:46.470276 24073 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.510779 24080 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:46.477319 24217 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.491489 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.456627 24075 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:46.495370 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:46.449780 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.518267 21379 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:46.507617 21335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 308) took 3353 ms (client timeout 2642 ms). Trace:
I20251215 08:59:46.609448 23851 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.562692 21974 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9.
W20251215 08:59:46.643893 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:46.481218 24142 consensus_meta.cc:287] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.634s user 0.011s sys 0.000s
I20251215 08:59:46.490141 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.573150 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:46.606417 21378 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:46.573292 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.656143 24009 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:46.526247 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.470494 23965 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:46.450781 21339 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:46.658946 24085 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:46.716998 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.494935 21259 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251215 08:59:46.741176 21408 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:46.730376 21263 raft_consensus.cc:1275] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20251215 08:59:46.738138 21264 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:46.742414 21263 pending_rounds.cc:85] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 4
I20251215 08:59:46.723893 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.730314 21975 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:46.724838 21266 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
W20251215 08:59:46.727069 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.731281 21406 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:46.717949 21258 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
W20251215 08:59:46.728052 21335 rpcz_store.cc:269] 1215 08:59:43.154519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.518005 (+363486us) service_pool.cc:225] Handling call
1215 08:59:46.507159 (+2989154us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:46.507399 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2502246784}
W20251215 08:59:46.745381 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.724779 21257 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:46.501128 21229 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:46.723022 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.747539 21415 raft_consensus.cc:2393] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:59:46.651742 24213 log.cc:927] Time spent T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
I20251215 08:59:46.726433 21260 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 9: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:46.437867 21412 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:46.727136 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 269290us
W20251215 08:59:46.675467 21307 connection.cc:232] RPC call timeout handler was delayed by -1.2695564890000242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
W20251215 08:59:46.680811 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:46.730826 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 124517us
I20251215 08:59:46.719520 21304 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:46.720873 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 319588us
I20251215 08:59:46.610780 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.493535 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:46.437870 21407 raft_consensus.cc:2368] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6.
I20251215 08:59:46.726145 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 289) took 3389 ms. Trace:
I20251215 08:59:46.743150 21263 pending_rounds.cc:107] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20251215 08:59:46.753705 21263 raft_consensus.cc:2889] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:46.748109 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:46.732920 21979 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 4 FOLLOWER]: Advancing to term 6
I20251215 08:59:46.583945 24161 raft_consensus.cc:3055] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:46.731014 23975 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:46.435635 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1802) took 1757 ms. Trace:
I20251215 08:59:46.438289 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.754624 21415 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251215 08:59:46.748624 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:46.496109 24164 consensus_meta.cc:287] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.574s user 0.010s sys 0.000s
I20251215 08:59:46.755298 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.749791 21265 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:46.756237 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.725878 24004 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.721045 24144 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251215 08:59:46.718480 24072 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.753607 21411 rpcz_store.cc:276] 1215 08:59:43.336996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.520052 (+2183056us) service_pool.cc:225] Handling call
1215 08:59:45.520525 (+ 473us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:45.583447 (+ 62922us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:45.583493 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:45.611369 (+ 27876us) log.cc:844] Serialized 26 byte log entry
1215 08:59:45.914661 (+303292us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:45.914728 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:45.914816 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:46.725951 (+811135us) raft_consensus.cc:1685] finished
1215 08:59:46.725973 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:46.726095 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42982,"raft.queue_time_us":51656,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":23315,"thread_start_us":283036,"threads_started":1,"wal-append.queue_time_us":497860}
I20251215 08:59:46.720009 23897 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:46.469714 24096 leader_election.cc:290] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.717191 24207 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:46.762707 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.782483 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.837147 21337 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:46.837386 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:46.837795 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:46.836100 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.839061 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.839407 24004 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.781749 24070 raft_consensus.cc:2804] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:46.727752 24073 raft_consensus.cc:2804] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:46.762213 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.762775 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:46.643271 21160 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45538 (request call id 152) took 11369 ms (client timeout 15000 ms). Trace:
I20251215 08:59:46.757431 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:46.610219 24218 log.cc:927] Time spent T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.144s user 0.003s sys 0.000s
I20251215 08:59:46.698956 24208 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.754724 23965 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.730006 23851 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:46.724555 24217 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.717479 24006 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.803571 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:46.811113 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 44113.8us
I20251215 08:59:46.795150 21379 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:46.448729 24196 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.401080 24215 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:46.861650 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 66584.5us
I20251215 08:59:46.825381 21414 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
I20251215 08:59:46.816514 21410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
W20251215 08:59:46.823596 21307 connection.cc:232] RPC call timeout handler was delayed by -1.1183564890001207s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
I20251215 08:59:46.832255 23975 raft_consensus.cc:697] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:46.832875 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.811575 24161 raft_consensus.cc:740] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:46.807883 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 143682us
I20251215 08:59:46.838022 21265 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:46.841117 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32802 (request call id 1558) took 2332 ms. Trace:
I20251215 08:59:46.799933 21979 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:46.791857 21381 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:46.837906 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:46.864310 21412 rpcz_store.cc:276] 1215 08:59:44.508324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.656243 (+1147919us) service_pool.cc:225] Handling call
1215 08:59:46.435827 (+779584us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:46.750394 (+314567us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:46.840900 (+ 90506us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54002f53d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:46.841066 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1791981440}
I20251215 08:59:46.865283 21257 raft_consensus.cc:2393] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:46.835386 21409 rpcz_store.cc:276] 1215 08:59:44.677905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:45.466846 (+788941us) service_pool.cc:225] Handling call
1215 08:59:46.435461 (+968615us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54002f53d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:46.435577 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402969344}
I20251215 08:59:46.807101 24144 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:46.725097 24075 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.861479 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:46.862759 21307 connection.cc:232] RPC call timeout handler was delayed by -1.0589564890000247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
W20251215 08:59:46.762547 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 261499us
I20251215 08:59:46.832751 21971 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
W20251215 08:59:46.863713 21230 leader_election.cc:336] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.865s (SENT)
I20251215 08:59:46.721410 24197 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.792707 24142 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:46.867606 21410 raft_consensus.cc:606] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:46.868681 21266 raft_consensus.cc:2410] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20251215 08:59:46.870137 21410 raft_consensus.cc:614] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:46.870513 21230 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:46.873941 21262 raft_consensus.cc:2393] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 4: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:46.872383 21379 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:46.874979 24072 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.875145 24006 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:46.872171 23965 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:46.876750 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 84975.2us
I20251215 08:59:46.877157 23872 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:46.878633 24075 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:46.879534 24208 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.883718 21973 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:46.868569 23932 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.885217 21258 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20251215 08:59:46.889797 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.870596 24073 raft_consensus.cc:697] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:46.911674 21337 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:46.892642 24215 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.893514 21232 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:46.856698 24207 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.894232 24197 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.876305 23975 consensus_queue.cc:237] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.856328 24070 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:46.867359 24004 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:46.903107 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:46.903859 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:46.868044 24144 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Advancing to term 10
W20251215 08:59:46.904819 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:46.905457 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:46.908113 21378 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:46.871424 24142 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:46.901154 23872 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:46.913107 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 56953.6us
I20251215 08:59:46.911427 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:46.875162 24217 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.911530 23901 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:46.928514 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:46.927361 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 55057.6us
I20251215 08:59:46.913893 23851 raft_consensus.cc:2804] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:46.924173 21335 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9.
I20251215 08:59:46.932292 21337 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:46.925794 23872 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:46.919881 24123 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:46.932840 21381 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:46.936141 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:46.936884 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 65454.4us
I20251215 08:59:46.921445 24004 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:46.940907 21379 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:46.921763 24070 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 1 FOLLOWER]: Advancing to term 2
W20251215 08:59:46.938592 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 66218.5us
I20251215 08:59:46.939260 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:46.943792 21381 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:46.945935 21973 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:46.941510 21378 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:46.944235 21333 raft_consensus.cc:2410] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:46.923646 24142 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:46.948321 21161 catalog_manager.cc:5654] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 2 to 3, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 3 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:46.952157 23984 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.952433 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:46.952867 23984 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:46.953617 23624 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:46.934934 24178 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:46.954066 23624 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:46.909307 24177 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20251215 08:59:46.939255 24208 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:46.957265 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:46.914139 23932 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:46.963506 21335 raft_consensus.cc:2393] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:46.935873 24197 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:46.969154 21337 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251215 08:59:46.944806 23901 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:46.974036 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:46.955232 24075 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:46.957899 23624 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:46.944804 24004 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:47.000615 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 56894.5us
I20251215 08:59:46.962496 24150 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:46.970324 24197 raft_consensus.cc:493] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:47.006474 24197 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:47.001646 24075 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.010290 24075 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:47.015066 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:46.962980 23851 raft_consensus.cc:697] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:46.963205 24123 raft_consensus.cc:697] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:47.019155 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 57899.2us
I20251215 08:59:47.019639 21410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:46.972191 24162 raft_consensus.cc:2764] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251215 08:59:47.003901 23901 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:47.013306 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 46705.3us
I20251215 08:59:47.024160 21410 raft_consensus.cc:606] T d8b3d79d7a014e568e385110044efb63 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:46.974964 24178 raft_consensus.cc:697] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.021076 24207 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:47.028060 21338 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:47.029368 21379 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:46.962622 24085 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.061s
I20251215 08:59:47.032444 24103 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:47.037528 21379 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:47.038630 24091 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:47.037544 24178 consensus_queue.cc:237] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.014098 23976 raft_consensus.cc:2804] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:47.040948 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:47.046977 21258 raft_consensus.cc:3060] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:47.047430 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:46.938371 24161 consensus_queue.cc:260] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.022293 24141 raft_consensus.cc:2764] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251215 08:59:47.050709 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:47.049801 21257 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:47.043980 21973 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:47.053639 21378 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:47.055539 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.030808 24196 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:47.041781 23965 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.046088 24091 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.071877 21261 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:47.075063 23846 raft_consensus.cc:2804] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:47.050861 23984 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:47.075764 24141 consensus_queue.cc:1059] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:47.077104 21381 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:47.077474 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.077414 23624 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.077975 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.078939 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.079049 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.079625 23624 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.079736 21413 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:47.080546 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:47.081087 21413 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32802 (request call id 1551) took 3551 ms (client timeout 1749 ms). Trace:
I20251215 08:59:47.081202 24162 consensus_queue.cc:1059] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:47.081689 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:47.081888 24162 consensus_queue.cc:1243] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer 0a2fcb89589c401d8efc8d86da4ba0f4 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20251215 08:59:47.081777 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.066249 23700 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:47.082731 24148 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.085896 23700 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.079903 24069 consensus_queue.cc:1048] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.006s
I20251215 08:59:47.086439 23700 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:47.086230 24148 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:47.051482 24150 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:47.085909 21339 raft_consensus.cc:2393] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.082405 21340 raft_consensus.cc:3060] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:47.079759 21336 raft_consensus.cc:2393] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.083562 21379 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:47.082252 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.079643 21341 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.081357 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.041947 23897 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.093663 21333 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:47.099640 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.099920 21332 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [4.3-6.5]
I20251215 08:59:47.099996 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:47.100239 21333 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9.
I20251215 08:59:47.100718 21261 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:47.101059 23974 raft_consensus.cc:3055] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:47.101127 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.101567 23974 raft_consensus.cc:740] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.101846 21333 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:47.102542 24150 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.104712 24150 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.094717 21337 raft_consensus.cc:1240] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:47.058904 23976 raft_consensus.cc:697] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.120877 23976 consensus_queue.cc:237] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.121798 21261 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:47.082054 24219 raft_consensus.cc:993] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: : Instructing follower 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:47.125084 24219 raft_consensus.cc:1081] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Signalling peer 0a2fcb89589c401d8efc8d86da4ba0f4 to start an election
I20251215 08:59:47.056978 24201 consensus_queue.cc:1048] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20251215 08:59:47.105149 23846 raft_consensus.cc:697] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.105545 24148 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.127565 23846 consensus_queue.cc:237] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.128312 24201 raft_consensus.cc:3055] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:47.105607 23965 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:47.129060 23965 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.129446 23965 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:47.130390 24070 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.074016 24091 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:47.081524 21413 rpcz_store.cc:269] 1215 08:59:43.529804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:43.639667 (+109863us) service_pool.cc:225] Handling call
1215 08:59:47.080850 (+3441183us) spinlock_profiling.cc:243] Waited 3.29 s on lock 0x7b540006ead4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:47.080988 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7569464192}
I20251215 08:59:47.106505 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:47.133339 24070 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:47.102499 23974 consensus_queue.cc:260] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.107579 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:47.139350 23974 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:47.141376 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.109153 23901 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:47.109486 23932 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.108881 21161 catalog_manager.cc:5654] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:47.110157 21334 raft_consensus.cc:2393] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.030428 24073 consensus_queue.cc:237] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.146059 21337 raft_consensus.cc:1240] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 2. Ops: []
I20251215 08:59:47.146615 21161 catalog_manager.cc:5654] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 2 to 4, leader changed from beefe15185ac42ef9a20251007f39a56 (127.19.163.65) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 4 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:47.110891 24215 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:47.113085 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:47.114256 21410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:47.051774 23872 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.119968 24104 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:47.132766 23897 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.151950 21379 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:47.200654 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 48832.3us
I20251215 08:59:47.201749 21379 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:47.128796 24201 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:47.187975 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.203275 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:47.204026 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.189301 21265 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
W20251215 08:59:47.193059 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 80047.9us
I20251215 08:59:47.193325 21258 raft_consensus.cc:2468] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
W20251215 08:59:47.197366 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 136283us
W20251215 08:59:47.182520 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.211761 21161 catalog_manager.cc:5654] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 3 to 5, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 5 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:47.217702 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:47.218287 21258 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:47.118019 24161 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:47.236663 21381 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:47.236914 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.238044 21337 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.238867 21161 catalog_manager.cc:5654] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 reported cstate change: term changed from 3 to 4, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66). New cstate: current_term: 4 leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:47.197192 24104 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.242327 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.202780 24150 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251215 08:59:47.279198 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.220500 24070 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.244783 21381 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:47.246013 23897 raft_consensus.cc:2804] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:47.252167 24177 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:47.253913 23901 consensus_queue.cc:1048] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:47.256340 24080 consensus_queue.cc:1059] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:47.257156 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.257102 23984 consensus_queue.cc:1059] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:47.198041 24215 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:47.260025 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.304852 21337 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:47.276520 21232 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:47.290724 21340 raft_consensus.cc:2468] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9.
I20251215 08:59:47.307006 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.241679 23976 raft_consensus.cc:2749] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:47.292966 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 48278.9us
I20251215 08:59:47.296556 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:47.191195 23851 consensus_queue.cc:237] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.311904 21336 raft_consensus.cc:2393] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.278548 24104 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:47.300361 23965 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.291023 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:47.317966 21338 raft_consensus.cc:2393] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.309417 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:47.310534 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.300508 23872 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.315675 21414 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:47.280038 24150 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.326071 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.316658 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:47.327529 21334 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:47.321262 21258 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:47.322285 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:47.292250 24009 consensus_meta.cc:287] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.561s user 0.010s sys 0.000s
I20251215 08:59:47.329694 21381 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:47.217345 24123 consensus_queue.cc:237] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:47.324146 24225 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.278s user 0.000s sys 0.001s
I20251215 08:59:47.300930 23978 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:47.297165 23897 raft_consensus.cc:697] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.347150 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:47.347435 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:47.350450 21335 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:47.296660 23700 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.296186 24070 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.295473 24197 raft_consensus.cc:515] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.387568 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.415650 21408 raft_consensus.cc:1240] T d69415d618114174b070bbc19a876119 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20251215 08:59:47.387841 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.363371 21415 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:47.323962 24064 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:47.351289 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.446641 21411 raft_consensus.cc:1275] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:47.362514 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:47.352334 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.489041 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:47.362805 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.482147 21407 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
W20251215 08:59:47.494177 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 197690us
I20251215 08:59:47.496284 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:47.324738 24201 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.499609 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.331558 24150 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:47.503885 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.317770 24219 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.500650 21334 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
W20251215 08:59:47.445374 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.506872 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.414927 24229 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251215 08:59:47.399919 24069 raft_consensus.cc:2804] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:47.512449 21229 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:47.524356 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.508471 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:47.519883 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.513226 21410 raft_consensus.cc:606] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:47.528424 21410 raft_consensus.cc:614] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:47.331076 24208 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.518527 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.523072 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:47.529657 21335 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
W20251215 08:59:47.519583 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.530316 21340 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:47.530705 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.518780 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:47.531275 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 254881us
I20251215 08:59:47.522202 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:47.485622 23901 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:47.511966 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.523671 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.533478 21334 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:47.530086 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.535144 21414 raft_consensus.cc:2393] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:47.531821 21415 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:47.531383 21971 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:47.532086 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:47.536036 21338 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:47.522755 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.536831 21334 raft_consensus.cc:1275] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20251215 08:59:47.536751 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.529932 21258 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 4.
I20251215 08:59:47.538633 21229 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:47.503401 24141 raft_consensus.cc:2764] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251215 08:59:47.536722 21411 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:47.532398 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.503196 24064 raft_consensus.cc:697] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.545372 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.537410 21334 pending_rounds.cc:85] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 5
I20251215 08:59:47.546193 21334 pending_rounds.cc:107] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20251215 08:59:47.472925 24075 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.546461 21334 raft_consensus.cc:2889] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:47.549837 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:47.550635 21258 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:47.545384 21266 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:47.389299 24144 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.549254 21340 raft_consensus.cc:1240] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:47.549161 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.553390 21335 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:47.527972 24229 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.552163 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.555657 24229 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:47.555424 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.556334 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.560488 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:59:47.407699 24237 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.152s
I20251215 08:59:47.529981 24207 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:47.561681 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.534896 24217 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:47.348467 24215 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.563127 23959 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.563629 21973 raft_consensus.cc:2393] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:47.543056 23846 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.577765 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 336) took 1084 ms. Trace:
I20251215 08:59:47.577807 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.583505 21413 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 10: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:47.578186 21264 rpcz_store.cc:276] 1215 08:59:46.493828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:46.838948 (+345120us) service_pool.cc:225] Handling call
1215 08:59:46.839325 (+ 377us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:46.839860 (+ 535us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:46.839891 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:46.840602 (+ 711us) log.cc:844] Serialized 52 byte log entry
1215 08:59:46.918860 (+ 78258us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:46.918892 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:46.918936 (+ 44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:47.577560 (+658624us) raft_consensus.cc:1685] finished
1215 08:59:47.577578 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:47.577708 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1399,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"thread_start_us":77926,"threads_started":1,"wal-append.queue_time_us":177316}
W20251215 08:59:47.585062 21944 leader_election.cc:336] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.642s (SENT)
I20251215 08:59:47.549441 24069 raft_consensus.cc:697] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.585772 21337 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 9: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:47.518821 24219 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.529558 23978 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.549623 23901 raft_consensus.cc:697] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:47.600762 21338 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:47.625128 21258 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:47.625586 21338 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251215 08:59:47.602049 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.570544 24217 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.571620 24227 raft_consensus.cc:3055] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:47.620733 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.622205 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.623485 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.626999 21339 raft_consensus.cc:2468] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 3.
I20251215 08:59:47.627038 21410 raft_consensus.cc:2368] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8.
I20251215 08:59:47.646615 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.647549 21976 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 9.
I20251215 08:59:47.523840 24091 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.628738 21381 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:47.537453 24208 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.657661 21415 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:47.631600 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:47.661280 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:47.662065 21258 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:47.632907 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.633667 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:47.634312 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:47.647145 21977 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:47.604750 21334 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b83855234bda47a1b06c3f932728766b"
dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:49734
I20251215 08:59:47.724313 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.723582 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:47.735033 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:47.649348 24004 consensus_meta.cc:287] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.625s user 0.007s sys 0.003s
W20251215 08:59:47.738293 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.575066 24207 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.742108 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:47.743072 21334 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20251215 08:59:47.650059 24217 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:47.748953 21334 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:47.750154 21406 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:47.628327 23965 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.645594 24227 raft_consensus.cc:740] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:47.756218 21971 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 9: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:47.636675 24201 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:47.757404 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.763794 21338 raft_consensus.cc:1240] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:47.764472 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:47.650902 24244 raft_consensus.cc:2804] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:47.724977 24177 raft_consensus.cc:2804] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:47.589356 23959 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.570132 23897 consensus_queue.cc:237] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.585049 23846 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.770262 21410 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:47.776646 21258 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:47.788021 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.789644 21340 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:47.792412 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.749363 24245 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:47.800071 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.800886 21407 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
W20251215 08:59:47.778394 24230 log.cc:927] Time spent T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20251215 08:59:47.820483 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:47.821734 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.822849 21304 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:47.779640 24197 leader_election.cc:290] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.753187 24004 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.753427 24229 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:47.832427 21946 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:47.837747 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.798074 24177 raft_consensus.cc:697] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:47.838519 21340 raft_consensus.cc:2468] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:47.839226 21307 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:47.865973 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:47.807277 24070 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:47.870729 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:47.874395 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:47.875303 21339 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:47.799546 24248 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:47.738020 24064 consensus_queue.cc:237] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.808866 24257 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.800086 24245 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:47.812788 24243 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:47.772991 24237 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.067s
I20251215 08:59:47.881047 24246 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:47.892710 24215 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:47.893558 24242 log.cc:927] Time spent T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.245s user 0.000s sys 0.001s
W20251215 08:59:47.894801 24161 consensus_meta.cc:287] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.660s user 0.010s sys 0.000s
I20251215 08:59:47.895588 24207 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:47.897526 24240 log.cc:927] Time spent T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
W20251215 08:59:47.880013 24234 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.389s user 0.000s sys 0.001s
I20251215 08:59:47.882284 24227 consensus_queue.cc:260] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.882525 24256 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:47.899192 24075 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:47.899581 24253 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:47.900404 24254 log.cc:927] Time spent T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20251215 08:59:47.902581 23978 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:47.903165 24238 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.332s user 0.000s sys 0.001s
I20251215 08:59:47.903852 24219 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:47.909698 24091 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:46.856350 21160 rpcz_store.cc:269] 1215 08:59:35.278426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:35.278822 (+ 396us) service_pool.cc:225] Handling call
1215 08:59:46.642951 (+11364129us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:44.845969 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:44.846605 (+ 636us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:44.846661 (+ 56us) write_op.cc:435] Acquired schema lock
1215 08:59:44.846695 (+ 34us) tablet.cc:660] Decoding operations
1215 08:59:44.847748 (+ 1053us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:44.847864 (+ 116us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:44.847903 (+ 39us) tablet.cc:683] Acquiring locks for 7 operations
1215 08:59:44.848699 (+ 796us) tablet.cc:699] Row locks acquired
1215 08:59:44.848738 (+ 39us) write_op.cc:260] PREPARE: finished
1215 08:59:44.848946 (+ 208us) write_op.cc:270] Start()
1215 08:59:44.852487 (+ 3541us) write_op.cc:276] Timestamp: P: 1765789184848862 usec, L: 0
1215 08:59:44.852533 (+ 46us) op_driver.cc:348] REPLICATION: starting
1215 08:59:44.853612 (+ 1079us) log.cc:844] Serialized 2835 byte log entry
1215 08:59:44.874528 (+ 20916us) op_driver.cc:464] REPLICATION: finished
1215 08:59:45.005332 (+130804us) write_op.cc:301] APPLY: starting
1215 08:59:45.005445 (+ 113us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:45.024302 (+ 18857us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:45.025511 (+ 1209us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:45.027631 (+ 2120us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:45.027931 (+ 300us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
1215 08:59:45.027975 (+ 44us) write_op.cc:312] APPLY: finished
1215 08:59:45.028315 (+ 340us) log.cc:844] Serialized 59 byte log entry
1215 08:59:45.028506 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:45.028622 (+ 116us) write_op.cc:454] Released schema lock
1215 08:59:45.028680 (+ 58us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:45.048812 (+ 20132us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:45.048845 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1521681,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":111729,"apply.run_cpu_time_us":3869,"apply.run_wall_time_us":43427,"num_ops":7,"prepare.queue_time_us":1373,"prepare.run_cpu_time_us":6493,"prepare.run_wall_time_us":15480,"raft.queue_time_us":8581,"raft.run_cpu_time_us":16684,"raft.run_wall_time_us":43179,"replication_time_us":21797,"thread_start_us":66952,"threads_started":4,"wal-append.queue_time_us":5366,"wal-append.run_cpu_time_us":5906,"wal-append.run_wall_time_us":1176262}]]}
I20251215 08:59:48.020011 21406 raft_consensus.cc:1240] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251215 08:59:47.911808 24229 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:47.912750 24208 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:47.913553 24257 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.806785 24069 consensus_queue.cc:237] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.915513 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:47.888321 24144 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:47.887972 24239 log.cc:927] Time spent T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.350s user 0.001s sys 0.000s
I20251215 08:59:48.000564 24258 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:48.037433 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 196030us
I20251215 08:59:47.934024 21334 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:47.884461 24070 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20251215 08:59:47.987962 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 408821us
I20251215 08:59:47.994539 24215 raft_consensus.cc:2749] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:47.929131 24245 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:48.049784 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:48.051301 21341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.004s sys 0.004s
W20251215 08:59:48.051606 21341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.004s sys 0.004s
W20251215 08:59:48.001596 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 162470us
W20251215 08:59:48.009450 24249 log.cc:927] Time spent T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.197s user 0.002s sys 0.001s
I20251215 08:59:48.014947 24177 consensus_queue.cc:237] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:48.016382 24142 consensus_meta.cc:287] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.017s user 0.006s sys 0.003s
I20251215 08:59:47.838210 23901 consensus_queue.cc:237] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:48.020524 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 186910us
W20251215 08:59:48.037211 23984 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.004s sys 0.004s
I20251215 08:59:47.990614 24256 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:47.890506 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:48.042690 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:48.064693 21258 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
W20251215 08:59:48.045557 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 226828us
I20251215 08:59:48.044649 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:48.066375 21259 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 10
I20251215 08:59:48.052557 21381 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:48.043041 23974 consensus_meta.cc:287] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.903s user 0.006s sys 0.008s
I20251215 08:59:48.055392 21304 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:48.027992 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:48.069669 21971 raft_consensus.cc:1275] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20251215 08:59:47.945940 24246 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:48.057392 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:48.081727 21260 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:48.018880 24150 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:48.059478 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:48.089669 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:48.089924 21307 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:48.061360 23984 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.004s sys 0.004s
I20251215 08:59:48.067348 21229 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:48.063035 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:48.026454 24244 raft_consensus.cc:697] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [term 9 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:48.071729 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 43863us
I20251215 08:59:48.074090 21336 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 2: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:48.044162 24070 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:48.088358 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:48.060232 21946 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:48.085544 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1851) took 1005 ms. Trace:
I20251215 08:59:48.032734 23978 consensus_queue.cc:1048] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20251215 08:59:48.116811 21232 leader_election.cc:400] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8.
I20251215 08:59:48.128584 21232 leader_election.cc:403] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:48.106232 21978 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20251215 08:59:48.117810 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 57651.4us
W20251215 08:59:48.127187 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 59918.5us
I20251215 08:59:48.106029 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:48.108345 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 48963.7us
I20251215 08:59:48.117664 21335 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:48.110260 21977 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 10: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:48.123539 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:48.117375 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:48.142097 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:48.154536 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:48.163760 21412 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:48.068801 24256 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:48.172757 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:48.176359 21336 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
I20251215 08:59:48.108630 24009 raft_consensus.cc:2804] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:48.108261 23959 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:48.125257 24004 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:48.034588 24253 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:48.127014 23974 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:48.118927 24215 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:48.126498 24070 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:48.177182 24207 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:48.065146 24142 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:48.123564 24201 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:48.380625 24267 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:48.381368 24267 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:48.381848 24267 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:48.114676 24260 consensus_queue.cc:1048] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.244s
I20251215 08:59:48.371379 24201 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:48.387789 24201 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:48.392673 24244 consensus_queue.cc:237] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:48.396539 23846 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:48.259677 23974 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:48.397535 23974 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:48.403244 23700 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:48.282604 24215 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:48.404322 24215 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:48.410800 24142 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:48.217613 21259 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 10.
W20251215 08:59:48.241627 21230 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.835s (SENT)
W20251215 08:59:48.242462 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 129553us
I20251215 08:59:48.080029 24236 consensus_queue.cc:1059] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.121s
W20251215 08:59:48.154155 24249 log.cc:927] Time spent T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20251215 08:59:48.199595 24104 consensus_meta.cc:287] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.868s user 0.011s sys 0.000s
I20251215 08:59:48.048738 24258 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:48.119181 24217 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:48.411724 21263 raft_consensus.cc:2393] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:48.416725 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 358262us
W20251215 08:59:48.426242 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 295061us
W20251215 08:59:48.426810 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 263005us
I20251215 08:59:48.427560 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:48.451189 24150 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:48.454631 24009 raft_consensus.cc:697] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:48.364868 21258 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:48.462373 21379 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:48.475988 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:48.458523 21230 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:48.472621 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:48.485090 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:48.411031 24257 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:48.458496 24104 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:48.492561 21971 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:48.494812 21378 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:48.498101 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:48.556658 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:48.557353 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 131914us
I20251215 08:59:48.558317 21232 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:48.487609 24236 consensus_queue.cc:1243] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer 195a56ac72a84018880bfb3bf6e6ef1d log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20251215 08:59:48.556560 21229 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:48.571910 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 77210.2us
I20251215 08:59:48.577337 21411 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:48.465502 24250 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:48.577363 21258 raft_consensus.cc:1275] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:48.584973 21339 raft_consensus.cc:1275] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251215 08:59:48.470165 24255 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8.
I20251215 08:59:48.496412 23897 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:48.589176 21232 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:48.593241 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 106468us
W20251215 08:59:48.595101 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 94892.3us
I20251215 08:59:48.594368 21259 raft_consensus.cc:1275] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:48.598171 21338 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251215 08:59:48.638474 21379 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:48.646983 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:48.578006 23851 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:48.577214 23959 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:48.650435 21944 leader_election.cc:336] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.751s (SENT)
W20251215 08:59:48.655299 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:48.656198 21307 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:48.488556 24247 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.141s
I20251215 08:59:48.630708 24268 consensus_queue.cc:1059] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20251215 08:59:48.588032 24236 raft_consensus.cc:3055] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Stepping down as leader of term 8
W20251215 08:59:48.661680 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 420070us
I20251215 08:59:48.649387 21265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2109) took 1431 ms. Trace:
I20251215 08:59:48.667415 21258 raft_consensus.cc:3055] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:48.680373 21258 raft_consensus.cc:740] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:48.681077 21258 consensus_queue.cc:260] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:48.682247 21258 raft_consensus.cc:3060] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:48.666683 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:48.671095 23984 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:48.666314 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 336) took 1063 ms. Trace:
I20251215 08:59:48.629446 24144 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:48.630086 23978 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:48.686569 21340 rpcz_store.cc:276] 1215 08:59:47.603371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.842712 (+239341us) service_pool.cc:225] Handling call
1215 08:59:47.888064 (+ 45352us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:47.937330 (+ 49266us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:47.937356 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:47.995461 (+ 58105us) log.cc:844] Serialized 26 byte log entry
1215 08:59:48.208945 (+213484us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:48.209007 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:48.209135 (+ 128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:48.666058 (+456923us) raft_consensus.cc:1685] finished
1215 08:59:48.666094 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:48.666251 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":213063,"threads_started":1,"wal-append.queue_time_us":651492}
I20251215 08:59:48.676167 21379 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:48.678943 21265 rpcz_store.cc:276] 1215 08:59:47.217690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.217866 (+ 176us) service_pool.cc:225] Handling call
1215 08:59:47.218170 (+ 304us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:47.222838 (+ 4668us) raft_consensus.cc:1534] Early marking committed up to index 4
1215 08:59:47.387488 (+164650us) log.cc:844] Serialized 14 byte log entry
1215 08:59:47.769618 (+382130us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:47.770174 (+ 556us) log.cc:844] Serialized 26 byte log entry
1215 08:59:47.770337 (+ 163us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:47.770384 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:47.770474 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:48.008702 (+238228us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:48.595488 (+586786us) raft_consensus.cc:1685] finished
1215 08:59:48.649208 (+ 53720us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:48.649328 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55239,"prepare.queue_time_us":274299,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":298827,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":49575,"spinlock_wait_cycles":7433088,"thread_start_us":337261,"threads_started":3,"wal-append.queue_time_us":317779}
I20251215 08:59:48.669649 21944 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:48.696267 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:48.645411 24142 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:48.849117 21230 leader_election.cc:336] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.690s (SENT)
W20251215 08:59:48.848697 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 192586us
I20251215 08:59:48.756816 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:48.864622 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 275557us
I20251215 08:59:48.118003 21341 rpcz_store.cc:276] 1215 08:59:47.079661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.106645 (+ 26984us) service_pool.cc:225] Handling call
1215 08:59:47.106922 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:47.107222 (+ 300us) raft_consensus.cc:1534] Early marking committed up to index 4
1215 08:59:47.312305 (+205083us) log.cc:844] Serialized 14 byte log entry
1215 08:59:48.084883 (+772578us) log.cc:844] Serialized 14 byte log entry
1215 08:59:48.085165 (+ 282us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:48.085207 (+ 42us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:48.085245 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:48.085331 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:48.085472 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":172137,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":41827,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":19665,"spinlock_wait_cycles":80926592,"thread_start_us":904857,"threads_started":2}
W20251215 08:59:48.853876 21264 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.011s sys 0.003s
I20251215 08:59:48.848248 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:48.677887 24245 consensus_meta.cc:287] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.616s user 0.011s sys 0.000s
W20251215 08:59:48.675832 24069 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.021s sys 0.026s
W20251215 08:59:48.851097 21335 consensus_meta.cc:287] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.697s user 0.007s sys 0.003s
I20251215 08:59:48.632120 24273 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:48.677281 24236 raft_consensus.cc:740] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:48.677038 23851 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:49.027385 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:48.887032 21232 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:49.141733 24255 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251215 08:59:49.141904 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 254948us
W20251215 08:59:49.112709 24070 consensus_meta.cc:287] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.741s user 0.010s sys 0.000s
I20251215 08:59:49.143314 21232 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:48.700269 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:48.711927 21263 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 2092) took 1956 ms (client timeout 2225 ms). Trace:
W20251215 08:59:49.155099 21263 rpcz_store.cc:269] 1215 08:59:46.755875 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:46.836536 (+ 80661us) service_pool.cc:225] Handling call
1215 08:59:48.711655 (+1875119us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400324454. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:48.711848 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3556414464}
W20251215 08:59:49.056738 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.157162 21336 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:48.904395 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:48.701907 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:48.962433 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:48.702914 21334 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:49.147356 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 471212us
I20251215 08:59:49.159761 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.148638 21260 raft_consensus.cc:2393] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 9: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:48.973363 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:48.909916 21259 raft_consensus.cc:1275] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251215 08:59:49.149838 21264 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.907s user 0.015s sys 0.011s
I20251215 08:59:49.145259 21335 raft_consensus.cc:2468] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:48.972657 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1823) took 1501 ms. Trace:
I20251215 08:59:49.162889 21411 rpcz_store.cc:276] 1215 08:59:47.470766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.542174 (+ 71408us) service_pool.cc:225] Handling call
1215 08:59:47.562799 (+ 20625us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:48.587688 (+1024889us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:48.972379 (+384691us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400340b54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:48.972581 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2319308160}
I20251215 08:59:49.006809 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 329) took 1688 ms. Trace:
I20251215 08:59:48.693784 24274 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:48.659915 23897 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:48.694326 24144 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:48.694012 24161 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:48.714174 24253 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.172291 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1904) took 1074 ms. Trace:
I20251215 08:59:49.184971 21335 rpcz_store.cc:276] 1215 08:59:48.087873 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:48.088026 (+ 153us) service_pool.cc:225] Handling call
1215 08:59:49.162640 (+1074614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:49.173179 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:49734 (request call id 1902) took 1871 ms. Trace:
I20251215 08:59:49.174024 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:49.167770 21381 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:48.688040 24246 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:48.661391 24104 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.178035 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 300) took 1863 ms. Trace:
W20251215 08:59:49.067602 24215 consensus_meta.cc:287] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.663s user 0.009s sys 0.000s
I20251215 08:59:49.167447 23978 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:49.144841 24069 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.020s user 0.023s sys 0.037s
I20251215 08:59:48.699700 24257 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:48.698694 24258 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:48.694751 24123 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:48.822722 24247 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:49.181725 24064 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20251215 08:59:49.183008 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:49.134277 24256 consensus_meta.cc:287] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.944s user 0.008s sys 0.002s
W20251215 08:59:48.886938 21230 leader_election.cc:336] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.843s (SENT)
I20251215 08:59:49.186208 21334 rpcz_store.cc:276] 1215 08:59:47.288079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.573045 (+284966us) service_pool.cc:225] Handling call
1215 08:59:48.701816 (+1128771us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b54001252d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bab6bd 00007f398cb6b800 00007f39861053b4 00007f3986105342 00007f3986105271 00007f39861051ec 00007f39861044a2 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:49.158938 (+457122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35744640}
I20251215 08:59:48.701630 24268 raft_consensus.cc:3055] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:49.063707 24217 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:48.902892 23846 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:49.188870 21412 rpcz_store.cc:276] 1215 08:59:47.314439 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.314592 (+ 153us) service_pool.cc:225] Handling call
1215 08:59:47.332610 (+ 18018us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:48.164366 (+831756us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:49.177761 (+1013395us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b540025fb54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:49.177973 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1910979840}
I20251215 08:59:48.677500 23959 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.167742 21337 rpcz_store.cc:276] 1215 08:59:47.318576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.638033 (+319457us) service_pool.cc:225] Handling call
1215 08:59:47.729419 (+ 91386us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:48.090857 (+361438us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:48.090904 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:48.110829 (+ 19925us) log.cc:844] Serialized 26 byte log entry
1215 08:59:48.254226 (+143397us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:48.254290 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:48.254391 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:49.006579 (+752188us) raft_consensus.cc:1685] finished
1215 08:59:49.006598 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:49.006732 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":72863,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":6319,"thread_start_us":475390,"threads_started":2,"wal-append.queue_time_us":881747}
I20251215 08:59:48.972868 24290 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:49.007144 24269 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:49.167191 24255 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:49.001690 24075 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:49.169521 24208 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:48.984669 24270 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:49.177590 24068 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:48.686019 23984 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.181525 24144 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.180241 23897 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:49.117655 24267 consensus_meta.cc:287] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.735s user 0.008s sys 0.003s
I20251215 08:59:48.849459 24201 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.219668 21230 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:48.729622 24009 consensus_queue.cc:237] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.196491 23978 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:49.176241 24245 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.199327 24104 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:49.174451 24070 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.198292 24246 raft_consensus.cc:2804] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:49.217176 24258 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:49.189657 24273 consensus_queue.cc:1059] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
W20251215 08:59:49.226486 24085 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.040s sys 0.004s
I20251215 08:59:49.218254 24257 raft_consensus.cc:697] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:49.217931 24064 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:49.255870 21971 raft_consensus.cc:1240] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:49.220832 24268 raft_consensus.cc:740] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:49.241396 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 590936us
I20251215 08:59:49.196329 24215 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:49.250993 23974 consensus_meta.cc:287] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.851s user 0.013s sys 0.000s
I20251215 08:59:49.203233 24247 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.250159 21946 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.198518 24274 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.266264 21333 raft_consensus.cc:1240] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 9. Current term is 10. Ops: []
W20251215 08:59:49.249308 24272 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.036s sys 0.088s
I20251215 08:59:49.224324 24256 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.200004 24123 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:49.266360 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.251729 24253 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:49.241820 24075 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.254609 24246 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:49.255900 24085 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s user 0.040s sys 0.006s
I20251215 08:59:49.257360 24064 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:49.284230 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:49.287518 24253 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.288086 24257 consensus_queue.cc:237] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.290071 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:49.260232 24269 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:49.244910 24208 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.247537 24267 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:49.300902 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 413971us
I20251215 08:59:49.227591 24236 consensus_queue.cc:260] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.280925 24123 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:49.273034 24272 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s user 0.037s sys 0.088s
I20251215 08:59:49.242734 24068 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.238348 24290 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:49.332180 21230 leader_election.cc:336] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.926s (SENT)
I20251215 08:59:49.334755 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.341136 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.343360 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 342) took 1525 ms. Trace:
I20251215 08:59:49.343923 21974 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:49.342324 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 168402us
I20251215 08:59:49.292762 24248 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:49.343886 21230 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:49.285228 24141 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Leader election won for term 10
I20251215 08:59:49.292124 24246 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:49.347543 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 443214us
I20251215 08:59:49.248054 24270 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.355448 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:49.353597 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 185890us
I20251215 08:59:49.266405 23974 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.344033 21264 rpcz_store.cc:276] 1215 08:59:47.817870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.818088 (+ 218us) service_pool.cc:225] Handling call
1215 08:59:47.872216 (+ 54128us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:47.881968 (+ 9752us) raft_consensus.cc:1534] Early marking committed up to index 9
1215 08:59:48.242896 (+360928us) log.cc:844] Serialized 14 byte log entry
1215 08:59:49.343009 (+1100113us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:49.343059 (+ 50us) raft_consensus.cc:1645] Marking committed up to 9
1215 08:59:49.343100 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:49.343187 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:49.343304 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17293,"prepare.queue_time_us":118995,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":64794,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":6362,"spinlock_wait_cycles":16791680,"thread_start_us":1396352,"threads_started":3,"wal-append.queue_time_us":987809}
W20251215 08:59:49.357966 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:49.358501 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.332964 23851 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.362577 24070 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.365904 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.344414 24255 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:49.334728 24269 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.374140 24297 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:49.364848 24290 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.378437 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.367136 24258 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:49.392645 21974 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:49.385412 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.396397 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.368247 24201 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.368525 24299 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:49.348202 24268 consensus_queue.cc:260] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.397550 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.409596 21979 raft_consensus.cc:2393] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:49.398764 24305 consensus_queue.cc:1048] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:49.381487 24215 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.383831 24247 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.379640 23984 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.401510 24244 raft_consensus.cc:3055] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:49.376189 23851 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251215 08:59:49.367974 24302 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:49.406148 24297 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.422005 24306 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20251215 08:59:49.382973 24256 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.408357 21336 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:49.409497 21263 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:49.386400 23959 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.380745 23974 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.379325 24267 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.442938 24297 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.408488 21381 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.443231 24064 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.430658 24144 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.384783 24075 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.395488 24085 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:49.384459 24270 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.417596 21379 leader_election.cc:304] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:49.452896 21944 leader_election.cc:336] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.942s (SENT)
I20251215 08:59:49.453430 21944 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:49.430401 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1912) took 1304 ms. Trace:
I20251215 08:59:49.437704 24248 raft_consensus.cc:697] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:49.438354 24141 raft_consensus.cc:697] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 10 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:49.413520 24251 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:49.441672 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 53301.8us
I20251215 08:59:49.411556 23846 consensus_queue.cc:1048] T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20251215 08:59:49.442559 24150 consensus_queue.cc:1048] T 067a0581bd0244539c1eca2c554e4619 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251215 08:59:49.418674 24244 raft_consensus.cc:740] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:49.445392 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.413198 24268 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.451368 21381 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.463497 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.454208 21336 rpcz_store.cc:276] 1215 08:59:48.126141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:48.699968 (+573827us) service_pool.cc:225] Handling call
1215 08:59:49.430326 (+730358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:49.408671 24299 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.458724 21307 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:49.449337 24004 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:49.461117 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:49.467792 21944 leader_election.cc:336] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.102s (SENT)
W20251215 08:59:49.467042 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 91384.2us
I20251215 08:59:49.468137 21412 raft_consensus.cc:2368] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
I20251215 08:59:49.453193 24245 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.468411 21944 leader_election.cc:304] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:49.457201 24075 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:49.457355 24161 raft_consensus.cc:2804] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:49.471521 21304 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:49.472159 21946 leader_election.cc:336] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.619s (SENT)
I20251215 08:59:49.474552 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:49.475106 21264 raft_consensus.cc:3060] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:49.474059 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.465559 24068 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.477824 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:49.483435 21415 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:49.478657 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.479202 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.479640 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.484262 21974 raft_consensus.cc:2368] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6.
I20251215 08:59:49.484468 21259 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:49.479779 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.454929 24085 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.480072 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:49.471915 24298 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251215 08:59:49.486912 21977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.487448 21977 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:49.471820 24251 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:49.488451 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:49.489415 21973 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:49.485581 21263 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:49.480950 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.481191 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:49.480787 24248 consensus_queue.cc:237] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.491549 21410 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:49.481587 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.482093 21413 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 9. Current term is 10. Ops: []
I20251215 08:59:49.482250 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:49.467850 24306 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:49.480319 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.486414 21412 raft_consensus.cc:1240] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:49.487784 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.492429 21414 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 10: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:49.494436 24253 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.495389 24313 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:49.495307 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.495981 24313 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:49.496384 24313 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.496316 21407 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:49.498836 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.499578 21408 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 5.
I20251215 08:59:49.500229 24247 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20251215 08:59:49.515065 21946 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.507843 21980 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:49.516038 21265 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
W20251215 08:59:49.521309 21230 leader_election.cc:336] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.645s (SENT)
W20251215 08:59:49.521889 21230 leader_election.cc:336] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.653s (SENT)
W20251215 08:59:49.523315 21230 leader_election.cc:336] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.660s (SENT)
W20251215 08:59:49.523978 21230 leader_election.cc:336] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.676s (SENT)
I20251215 08:59:49.529688 24269 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.506502 21410 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:49.524763 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.526160 21229 leader_election.cc:400] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Vote denied by peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831) with higher term. Message: Invalid argument: T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6.
I20251215 08:59:49.531693 21229 leader_election.cc:403] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251215 08:59:49.475448 24064 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.527962 21307 leader_election.cc:400] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8.
I20251215 08:59:49.532582 21307 leader_election.cc:403] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20251215 08:59:49.533269 24251 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8.
I20251215 08:59:49.533787 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.535858 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:49.477798 24144 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.486483 24208 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.483098 24161 raft_consensus.cc:697] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:49.485997 24257 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
W20251215 08:59:49.573868 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 51231.2us
I20251215 08:59:49.577950 21229 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.566605 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:49.580552 21410 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:49.553552 21230 leader_election.cc:336] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.963s (SENT)
I20251215 08:59:49.582386 21230 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:49.555650 21415 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:49.525916 24256 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:49.589761 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.599606 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.619362 21307 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:49.624064 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:49.624761 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.573608 24251 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:49.638451 21410 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:49.640398 24250 raft_consensus.cc:2749] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:49.630451 23984 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:49.514446 24141 consensus_queue.cc:237] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.641448 23984 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:49.643764 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 59347us
I20251215 08:59:49.644564 24258 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:49.528373 24244 consensus_queue.cc:260] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.645213 24258 raft_consensus.cc:697] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:49.645818 24244 raft_consensus.cc:3060] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.646677 24258 consensus_queue.cc:237] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.629598 24257 raft_consensus.cc:697] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:49.536509 24274 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.620563 24256 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:49.657958 24161 consensus_queue.cc:237] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.659662 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.579538 24215 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20251215 08:59:49.621166 24269 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:49.660239 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 106754us
W20251215 08:59:49.715131 21230 leader_election.cc:336] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.123s (SENT)
I20251215 08:59:49.715610 21230 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:49.663429 21381 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:49.658566 24299 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.633033 24302 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
W20251215 08:59:49.650288 24295 log.cc:927] Time spent T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.294s user 0.002s sys 0.000s
I20251215 08:59:49.666504 23984 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:49.710799 21258 pending_rounds.cc:85] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 4
I20251215 08:59:49.729390 21258 pending_rounds.cc:107] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20251215 08:59:49.729775 21258 raft_consensus.cc:2889] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:49.719588 24269 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.730399 21973 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251215 08:59:49.711149 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.693459 23978 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.712107 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:49.717047 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 53712.4us
I20251215 08:59:49.723824 21264 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:49.733690 24270 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831) with higher term. Message: Invalid argument: T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6.
I20251215 08:59:49.661059 24256 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.734218 21381 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:49.733619 21980 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:49.737273 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:49.712785 24219 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:49.738919 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:49.740338 21946 leader_election.cc:336] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 2.186s (SENT)
I20251215 08:59:49.738943 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:49.740865 21946 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:49.741914 21230 leader_election.cc:336] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.157s (SENT)
I20251215 08:59:49.742461 21230 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.744334 24227 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:49.745790 21408 raft_consensus.cc:1240] T 9b36e018bdd140648e80e809a47269c8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20251215 08:59:49.748590 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.745159 21307 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.726341 23974 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:49.760464 21414 raft_consensus.cc:1240] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20251215 08:59:49.795064 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 53192.4us
W20251215 08:59:49.796255 21230 leader_election.cc:336] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.804s (SENT)
I20251215 08:59:49.797099 21414 raft_consensus.cc:1240] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251215 08:59:49.650262 24257 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:49.798548 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:49.849924 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:49.812338 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:49.849756 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 51670.9us
W20251215 08:59:49.854264 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 115946us
I20251215 08:59:49.739637 24245 raft_consensus.cc:2749] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 08:59:49.855062 21307 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:49.733342 24269 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.866724 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.853773 24208 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251215 08:59:49.872540 24208 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:49.872920 24208 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:49.874200 23978 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:49.874334 21263 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251215 08:59:49.794499 24201 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20251215 08:59:49.876236 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.879910 24085 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:49.881320 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.882449 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:49.882951 21412 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:49.886583 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.886775 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:49.888119 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:49.888947 21332 raft_consensus.cc:2410] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 9 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251215 08:59:49.889104 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:49.889451 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 149137us
I20251215 08:59:49.889642 21336 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:49.889837 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:49.890426 21339 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.890687 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:49.891408 21337 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:49.891517 21946 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.893374 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:49.887694 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.852236 21229 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.735982 24302 raft_consensus.cc:697] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:49.727006 23897 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:49.757288 23984 raft_consensus.cc:493] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:49.852502 24303 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
I20251215 08:59:49.906363 21341 raft_consensus.cc:3060] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.757580 24219 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.907202 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:49.940397 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:49.918272 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:49.916083 21339 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:49.921757 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:49.763828 23974 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:49.916553 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 64527.1us
I20251215 08:59:49.752084 24258 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251215 08:59:49.928930 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.930373 21946 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.917927 24004 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20251215 08:59:49.799947 24253 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:49.935189 21340 raft_consensus.cc:3055] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:49.936323 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:49.953378 21334 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:49.952908 21340 raft_consensus.cc:740] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:49.932495 21232 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:49.750867 24069 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:49.942830 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:49.961117 21336 raft_consensus.cc:2410] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251215 08:59:49.941258 21414 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 4.
I20251215 08:59:49.951043 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:49.927942 24304 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader election won for term 4
I20251215 08:59:49.959712 21946 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:49.954847 21340 consensus_queue.cc:260] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.924654 24292 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:49.975481 21415 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:49.956161 21412 raft_consensus.cc:2468] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:49.927474 24085 raft_consensus.cc:2804] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:49.968555 21414 raft_consensus.cc:1275] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251215 08:59:49.968441 21232 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:49.926432 23978 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
W20251215 08:59:49.973189 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:49.951551 23974 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:49.967512 24274 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:49.929044 24201 consensus_queue.cc:1059] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20251215 08:59:49.909723 24250 consensus_queue.cc:1059] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20251215 08:59:49.980245 24268 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:49.984700 24298 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:49.952071 23984 raft_consensus.cc:515] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:49.990335 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:49.984956 24304 raft_consensus.cc:697] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:49.960415 24069 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:49.926733 24299 consensus_queue.cc:1059] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
I20251215 08:59:49.956096 24258 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:49.965850 24269 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:49.969309 24251 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:50.004374 24253 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:49.989841 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.006763 21334 raft_consensus.cc:2468] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:49.980870 24292 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:49.972250 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:49.984411 24312 consensus_queue.cc:1059] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20251215 08:59:50.012269 21336 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:49.992656 21340 raft_consensus.cc:3060] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:49.965561 24313 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:50.059758 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 70012.4us
I20251215 08:59:50.077447 21232 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:50.052165 24320 log.cc:927] Time spent T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20251215 08:59:50.022270 24298 raft_consensus.cc:697] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:50.049379 24069 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:50.007458 24274 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.143415 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.149415 21263 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 8
I20251215 08:59:50.066910 24292 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:50.087318 24319 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.371s user 0.000s sys 0.001s
I20251215 08:59:50.057479 24258 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:50.056936 24269 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:50.056740 24251 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:50.145340 24299 raft_consensus.cc:3055] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Stepping down as leader of term 8
W20251215 08:59:50.173210 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 95870.4us
I20251215 08:59:50.174363 21232 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.151482 24253 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:50.154505 21336 raft_consensus.cc:3055] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:50.014046 24208 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:50.153589 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 281350us
W20251215 08:59:50.180629 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:50.153358 24085 raft_consensus.cc:697] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:50.183777 21336 raft_consensus.cc:740] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:50.195953 21232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:50.167481 24326 log.cc:927] Time spent T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
I20251215 08:59:50.196527 21336 consensus_queue.cc:260] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.200021 21307 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:50.163403 24251 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:50.163291 24269 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:50.208501 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.209371 21266 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 4.
W20251215 08:59:50.194133 24123 consensus_meta.cc:287] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.849s user 0.013s sys 0.000s
I20251215 08:59:50.212229 21336 raft_consensus.cc:3060] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:50.181334 24299 raft_consensus.cc:740] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:50.215346 21381 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:50.224959 21339 raft_consensus.cc:2393] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 10: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:50.198920 23897 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:50.202003 24253 raft_consensus.cc:697] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:50.229110 24244 consensus_meta.cc:287] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.583s user 0.008s sys 0.004s
I20251215 08:59:50.099349 24302 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.267261 21337 raft_consensus.cc:2410] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:50.201283 24215 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
I20251215 08:59:50.291832 21336 raft_consensus.cc:1275] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251215 08:59:50.227437 24123 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:50.295445 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 99896.3us
I20251215 08:59:50.263783 24219 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:50.257936 24339 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251215 08:59:50.310281 21265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.004s sys 0.015s
W20251215 08:59:50.310570 21265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s user 0.004s sys 0.015s
I20251215 08:59:50.314316 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.315142 21307 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.317633 21266 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20251215 08:59:50.295250 24345 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:50.324401 21341 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:50.204249 24304 consensus_queue.cc:237] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.313860 24346 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251215 08:59:50.259847 24298 consensus_queue.cc:237] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.348737 21263 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
W20251215 08:59:50.353760 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:50.334040 24327 log.cc:927] Time spent T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.309s user 0.000s sys 0.001s
W20251215 08:59:50.381407 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 66310.9us
I20251215 08:59:50.382035 24354 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.382851 24354 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.384411 21379 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.385397 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.386103 21414 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:50.385973 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.386989 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.387430 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.387670 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:50.387998 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.388445 21413 raft_consensus.cc:2410] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:50.389083 23846 raft_consensus.cc:2804] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:50.389357 21262 raft_consensus.cc:1240] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:50.389606 23846 raft_consensus.cc:697] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:50.389804 24345 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:50.390585 23846 consensus_queue.cc:237] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.392148 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.393199 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.395128 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.395881 21263 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:50.301788 24085 consensus_queue.cc:237] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.327883 23984 leader_election.cc:290] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:50.361794 24328 log.cc:927] Time spent T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.238s user 0.002s sys 0.000s
W20251215 08:59:50.347826 24246 consensus_meta.cc:287] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.981s user 0.009s sys 0.004s
I20251215 08:59:50.334231 24313 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.404292 21379 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
W20251215 08:59:50.345099 24331 log.cc:927] Time spent T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20251215 08:59:50.387125 21414 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:50.454110 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.402990 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.454663 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.455405 21412 raft_consensus.cc:2393] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 9: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:50.350204 24335 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:50.354373 24346 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:50.396852 24330 log.cc:927] Time spent T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.231s user 0.000s sys 0.003s
I20251215 08:59:50.454183 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.459132 21262 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:50.458650 21381 leader_election.cc:304] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:50.338927 24312 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20251215 08:59:50.317142 24299 consensus_queue.cc:260] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:50.408521 24340 log.cc:927] Time spent T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.125s user 0.000s sys 0.002s
I20251215 08:59:50.475898 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.476678 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.476783 21262 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
W20251215 08:59:50.457417 24350 log.cc:927] Time spent T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20251215 08:59:50.475529 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.481932 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:50.481261 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 77036.9us
I20251215 08:59:50.457825 24208 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:50.484432 21262 raft_consensus.cc:2393] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:50.487007 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.487586 21413 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 8.
I20251215 08:59:50.488524 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.490952 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:50.491700 21413 raft_consensus.cc:3055] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Stepping down as leader of term 8
I20251215 08:59:50.492282 21413 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:50.494730 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.497262 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:50.497346 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:50.468459 24346 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:50.504889 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.502774 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.494029 21413 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.506906 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.506910 21413 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:50.507684 21262 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:50.475497 24177 raft_consensus.cc:2804] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:50.501835 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.471894 24335 raft_consensus.cc:697] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:50.452740 24246 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.509765 21263 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:50.380548 24253 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.499737 24245 consensus_queue.cc:1059] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
W20251215 08:59:50.500281 24227 consensus_meta.cc:287] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.751s user 0.011s sys 0.000s
I20251215 08:59:50.517825 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:50.486239 24274 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:50.533577 21412 raft_consensus.cc:2393] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:50.536427 24196 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:50.536793 24332 log.cc:927] Time spent T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.201s user 0.000s sys 0.002s
I20251215 08:59:50.543187 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.545423 21263 raft_consensus.cc:1240] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
W20251215 08:59:50.555608 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 78433.3us
I20251215 08:59:50.548321 24141 raft_consensus.cc:3055] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Stepping down as leader of term 9
I20251215 08:59:50.554332 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:50.563258 21413 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 9.
W20251215 08:59:50.570719 21307 connection.cc:232] RPC call timeout handler was delayed by -3.1828340399999888s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20251215 08:59:50.571336 21307 connection.cc:232] RPC call timeout handler was delayed by -3.1171340399999963s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20251215 08:59:50.571714 21307 connection.cc:232] RPC call timeout handler was delayed by -3.0784340400000474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
W20251215 08:59:50.572117 21307 connection.cc:232] RPC call timeout handler was delayed by -3.0694340400000328s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20251215 08:59:50.572422 21307 connection.cc:232] RPC call timeout handler was delayed by -2.9740340399999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
W20251215 08:59:50.572782 21307 connection.cc:232] RPC call timeout handler was delayed by -2.8210340399999723s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1902s for a response
W20251215 08:59:50.577214 21307 connection.cc:232] RPC call timeout handler was delayed by -2.7247340400000439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
W20251215 08:59:50.577945 21307 connection.cc:232] RPC call timeout handler was delayed by -2.5996340399999553s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
W20251215 08:59:50.578523 21307 connection.cc:232] RPC call timeout handler was delayed by -2.5645340400000123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
W20251215 08:59:50.579111 21307 connection.cc:232] RPC call timeout handler was delayed by -2.2963340400000334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2485s for a response
W20251215 08:59:50.579563 21307 connection.cc:232] RPC call timeout handler was delayed by -1.9147340399999848s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2909s for a response
W20251215 08:59:50.579923 21307 connection.cc:232] RPC call timeout handler was delayed by -1.6888340399999606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31600000000000006s for a response
I20251215 08:59:50.581355 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.590065 24362 consensus_queue.cc:1059] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251215 08:59:50.590893 24177 raft_consensus.cc:697] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:50.591907 24177 consensus_queue.cc:237] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.601881 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.501631 24342 consensus_queue.cc:1059] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20251215 08:59:50.607220 24257 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:50.573812 24208 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.569360 24141 raft_consensus.cc:740] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:50.609783 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.551899 24196 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.613245 24360 consensus_queue.cc:1048] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.064s
I20251215 08:59:50.562489 24312 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20251215 08:59:50.586297 24215 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:50.629341 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 140215us
I20251215 08:59:50.633180 21307 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:50.633180 21230 leader_election.cc:336] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.866s (SENT)
W20251215 08:59:50.634047 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 85377.6us
I20251215 08:59:50.634239 21230 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:50.635952 21980 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:50.638304 21307 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:50.626777 24256 consensus_meta.cc:287] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.892s user 0.011s sys 0.001s
W20251215 08:59:50.641774 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:50.646947 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.647085 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.630074 24304 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:50.641079 24256 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.660773 21412 raft_consensus.cc:1240] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20251215 08:59:50.662750 21413 raft_consensus.cc:1240] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:50.642880 24257 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251215 08:59:50.630689 24354 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:50.643464 24348 raft_consensus.cc:2804] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:50.643348 24227 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.671087 21413 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: [6.5-7.6]
I20251215 08:59:50.626878 24208 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.687044 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:50.687762 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.688271 21412 raft_consensus.cc:2393] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 10: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:50.688809 21265 raft_consensus.cc:2393] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:50.692142 24269 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.653851 24215 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:50.693411 24246 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:50.694105 24348 raft_consensus.cc:697] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:50.693995 24347 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20251215 08:59:50.695056 24348 consensus_queue.cc:237] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:50.698421 21262 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 9.
I20251215 08:59:50.643381 24335 consensus_queue.cc:237] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:50.673625 24304 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251215 08:59:50.722608 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 139318us
I20251215 08:59:50.731333 24251 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.734635 24208 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:50.702979 24227 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:50.735299 23974 consensus_meta.cc:287] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.728s user 0.009s sys 0.000s
I20251215 08:59:50.737139 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:50.745738 21980 raft_consensus.cc:2393] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
W20251215 08:59:50.737254 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 90741.7us
I20251215 08:59:50.686877 24141 consensus_queue.cc:260] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.724713 24257 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.746483 23974 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.761430 24251 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.732251 24346 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.779168 24346 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
W20251215 08:59:50.779350 24069 consensus_meta.cc:287] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.621s user 0.001s sys 0.008s
I20251215 08:59:50.779772 24069 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.743747 21971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46170 (request call id 536) took 1861 ms. Trace:
I20251215 08:59:50.747296 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:50.762012 24141 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:50.762696 24256 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:50.763022 24227 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.753885 24341 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251215 08:59:50.741923 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 258719us
I20251215 08:59:50.774956 21307 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.776319 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:50.790378 21407 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:50.790983 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1842) took 1093 ms. Trace:
I20251215 08:59:50.791337 21414 raft_consensus.cc:2368] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7.
I20251215 08:59:50.783568 21974 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:50.782375 21971 rpcz_store.cc:276] 1215 08:59:48.875684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:48.886332 (+ 10648us) service_pool.cc:225] Handling call
1215 08:59:48.947411 (+ 61079us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:49.256231 (+308820us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:50.736401 (+1480170us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b540064cb54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:50.736557 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":708975872}
I20251215 08:59:50.789511 21379 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.810408 24251 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:50.819718 24346 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:50.791383 21407 rpcz_store.cc:276] 1215 08:59:49.697735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:50.764658 (+1066923us) service_pool.cc:225] Handling call
1215 08:59:50.790916 (+ 26258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:50.781481 24347 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
W20251215 08:59:50.820915 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 46095.2us
I20251215 08:59:50.781980 24342 raft_consensus.cc:493] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.821769 21379 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.823247 21307 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.825553 21379 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:50.822292 24292 consensus_meta.cc:287] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.658s user 0.011s sys 0.000s
I20251215 08:59:50.825969 21307 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.827684 21379 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.827769 21307 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.785609 24257 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.831561 21307 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.832369 21379 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.834263 21307 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.829573 24103 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:50.826740 24346 raft_consensus.cc:2804] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:50.826591 24251 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:50.830614 24150 raft_consensus.cc:2749] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:50.829633 24208 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:50.835460 24215 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 08:59:50.832167 24304 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:50.833261 24069 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:50.864820 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:50.844708 21232 leader_election.cc:336] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.660s (SENT)
I20251215 08:59:50.865744 21265 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:50.864540 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.867100 21232 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:50.867417 21263 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 6
W20251215 08:59:50.869513 21230 leader_election.cc:336] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.118s (SENT)
W20251215 08:59:50.872200 24258 consensus_meta.cc:287] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.690s user 0.010s sys 0.001s
I20251215 08:59:50.872704 24258 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.874504 24258 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:50.876168 24322 log.cc:927] Time spent T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.991s user 0.006s sys 0.003s
I20251215 08:59:50.890715 21263 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:50.883602 21975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 673) took 2405 ms. Trace:
I20251215 08:59:50.883662 24257 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.892398 24341 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:50.892122 21975 rpcz_store.cc:276] 1215 08:59:48.471871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:48.471990 (+ 119us) service_pool.cc:225] Handling call
1215 08:59:48.563473 (+ 91483us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:48.676353 (+112880us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:48.676384 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:49.422178 (+745794us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b2c000472a8. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:49.422630 (+ 452us) log.cc:844] Serialized 26 byte log entry
1215 08:59:49.581248 (+158618us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:49.581291 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:49.581345 (+ 54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:50.844777 (+1263432us) spinlock_profiling.cc:243] Waited 9.23 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:50.877184 (+ 32407us) raft_consensus.cc:1685] finished
1215 08:59:50.877207 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:50.877361 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":32205,"raft.run_cpu_time_us":579,"raft.run_wall_time_us":13653,"spinlock_wait_cycles":45816960,"thread_start_us":220046,"threads_started":2,"wal-append.queue_time_us":293088}
I20251215 08:59:50.829572 24292 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.865803 24103 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20251215 08:59:50.910804 21230 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.553s (SENT)
I20251215 08:59:50.869740 24251 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:50.869565 24269 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.869614 24346 raft_consensus.cc:697] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:50.915741 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.916239 21258 raft_consensus.cc:1240] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:50.913036 21265 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
W20251215 08:59:50.914494 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 80047.3us
I20251215 08:59:50.866639 24347 consensus_queue.cc:1059] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20251215 08:59:50.829046 24342 raft_consensus.cc:515] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.913594 24251 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:50.921855 24364 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.924014 24380 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:50.917580 23974 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:50.930387 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:50.931334 24346 consensus_queue.cc:237] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.905318 23846 consensus_queue.cc:1048] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20251215 08:59:50.932392 24363 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.919785 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:50.926735 24372 raft_consensus.cc:3055] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 08:59:50.942625 21229 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.944644 24372 raft_consensus.cc:740] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:50.948096 24372 consensus_queue.cc:260] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.950592 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 311) took 1470 ms. Trace:
I20251215 08:59:50.950855 24342 leader_election.cc:290] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.951463 24064 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:50.951958 24064 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.953241 24345 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:50.954370 24064 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:50.955567 24292 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:50.950978 21406 rpcz_store.cc:276] 1215 08:59:49.479873 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.481109 (+ 1236us) service_pool.cc:225] Handling call
1215 08:59:49.481451 (+ 342us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:49.500490 (+ 19039us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:49.500526 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:49.501160 (+ 634us) log.cc:844] Serialized 52 byte log entry
1215 08:59:49.663100 (+161940us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:49.663157 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:50.950216 (+1287059us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b54001757d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:50.950261 (+ 45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:50.950308 (+ 47us) raft_consensus.cc:1685] finished
1215 08:59:50.950321 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:50.950535 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5492,"raft.queue_time_us":6661,"spinlock_wait_cycles":51277952,"thread_start_us":161539,"threads_started":1,"wal-append.queue_time_us":224642,"wal-append.run_cpu_time_us":1844,"wal-append.run_wall_time_us":1147629}
I20251215 08:59:50.942677 21265 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:50.946678 24363 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:50.938339 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 75831us
I20251215 08:59:50.938848 24251 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:50.958318 24363 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:50.954593 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:50.962143 21379 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:50.942688 21307 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:50.965042 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 43524.5us
I20251215 08:59:50.966192 21307 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:50.967506 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:50.967639 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:50.968683 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:50.971166 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:50.971203 21265 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:50.973572 24390 consensus_queue.cc:1059] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20251215 08:59:50.973835 21381 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:50.974500 24390 raft_consensus.cc:3055] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:50.974855 24390 raft_consensus.cc:740] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:50.974893 23846 raft_consensus.cc:2804] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader election won for term 6
I20251215 08:59:50.975181 21381 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:50.975548 23846 raft_consensus.cc:697] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:50.976202 24396 raft_consensus.cc:2804] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:50.976771 24396 raft_consensus.cc:697] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:50.977739 24377 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20251215 08:59:50.977654 24396 consensus_queue.cc:237] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.979043 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1843) took 1268 ms. Trace:
I20251215 08:59:50.975734 24390 consensus_queue.cc:260] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.979373 21410 rpcz_store.cc:276] 1215 08:59:49.711185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.711312 (+ 127us) service_pool.cc:225] Handling call
1215 08:59:49.711698 (+ 386us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:49.720702 (+ 9004us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:49.720729 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:50.538788 (+818059us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b2c00092518. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f39864b29b1 00007f39864c1fe2 00007f3987bef851 00007f3987bb40ba 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22
1215 08:59:50.549713 (+ 10925us) log.cc:844] Serialized 26 byte log entry
1215 08:59:50.646372 (+ 96659us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:50.646404 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:50.646464 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:50.978731 (+332267us) raft_consensus.cc:1685] finished
1215 08:59:50.978749 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:50.978966 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9563,"raft.queue_time_us":8984,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":10531,"spinlock_wait_cycles":13112576,"thread_start_us":62671,"threads_started":1,"wal-append.queue_time_us":262653}
I20251215 08:59:50.976516 23846 consensus_queue.cc:237] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.980486 24390 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:50.941244 24380 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:50.983136 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1930) took 1057 ms. Trace:
I20251215 08:59:50.983613 24380 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.983529 21339 rpcz_store.cc:276] 1215 08:59:49.926105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.950713 (+ 24608us) service_pool.cc:225] Handling call
1215 08:59:50.982954 (+1032241us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b540009c4d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:50.983073 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":602023552}
I20251215 08:59:50.984755 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1927) took 1063 ms. Trace:
I20251215 08:59:50.985183 21335 rpcz_store.cc:276] 1215 08:59:49.921237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.921396 (+ 159us) service_pool.cc:225] Handling call
1215 08:59:50.984568 (+1063172us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b5400048154. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:50.984685 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1402983808}
I20251215 08:59:50.986287 24304 consensus_queue.cc:1059] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:50.986639 21381 leader_election.cc:304] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:50.987262 24398 raft_consensus.cc:3055] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:50.941058 24364 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.987653 24398 raft_consensus.cc:740] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:50.988519 24398 consensus_queue.cc:260] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.989349 24364 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:50.989992 23974 raft_consensus.cc:2804] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:50.990003 24247 raft_consensus.cc:2804] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:50.990581 23974 raft_consensus.cc:697] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:50.991549 24398 raft_consensus.cc:3060] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:50.991642 23974 consensus_queue.cc:237] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:50.990026 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 347) took 1623 ms. Trace:
I20251215 08:59:50.994073 21260 rpcz_store.cc:276] 1215 08:59:49.366752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.367706 (+ 954us) service_pool.cc:225] Handling call
1215 08:59:49.367949 (+ 243us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:49.371396 (+ 3447us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:49.371431 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:49.371478 (+ 47us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:49.371516 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:50.989800 (+1618284us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b5400170054. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:50.989843 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:50.989962 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7229184}
I20251215 08:59:50.995605 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:50.996297 21260 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:50.990651 24247 raft_consensus.cc:697] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:50.990453 21409 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1818) took 3651 ms (client timeout 1645 ms). Trace:
W20251215 08:59:50.997923 21409 rpcz_store.cc:269] 1215 08:59:47.339046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:47.399787 (+ 60741us) service_pool.cc:225] Handling call
1215 08:59:47.555223 (+155436us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:50.990270 (+3435047us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54001b8ad4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:50.990387 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3787028992}
I20251215 08:59:51.000363 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:51.000691 21232 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:50.999521 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:51.002920 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.003369 21232 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:51.003672 21335 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:51.004405 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.005874 21332 raft_consensus.cc:3060] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:51.006052 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.007963 21232 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:51.020138 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:51.020999 21232 leader_election.cc:336] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.756s (SENT)
I20251215 08:59:51.021494 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:51.009528 21381 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:51.021718 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.023064 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:51.023342 21412 raft_consensus.cc:1240] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251215 08:59:51.023626 21409 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:51.024875 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.027482 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.028020 21339 raft_consensus.cc:2393] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:51.028520 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.029248 21406 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:51.029139 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.029994 21337 raft_consensus.cc:2376] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9. Re-sending same reply.
I20251215 08:59:51.033316 21407 raft_consensus.cc:1240] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 9. Current term is 10. Ops: []
I20251215 08:59:50.969769 24347 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:51.036686 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:50.998695 24208 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:51.040388 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.025364 23984 raft_consensus.cc:2804] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:51.035989 24245 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:51.009558 24064 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:51.036710 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.009430 24339 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:51.019372 24364 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251215 08:59:51.038482 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.011663 24363 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:51.041986 24336 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:51.019163 24335 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:51.044538 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.056198 21335 raft_consensus.cc:2368] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6.
I20251215 08:59:51.056566 21336 raft_consensus.cc:2393] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:51.060916 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:51.062161 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.063261 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.064234 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.028566 24345 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20251215 08:59:51.072392 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:51.074254 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:51.077045 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:51.077952 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.078181 21412 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20251215 08:59:51.084259 24281 log.cc:927] Time spent T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 2.131s user 0.000s sys 0.015s
I20251215 08:59:51.057499 24336 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:51.085552 24336 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:51.086407 21979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 705) took 1797 ms (client timeout 1997 ms). Trace:
W20251215 08:59:51.086846 21979 rpcz_store.cc:269] 1215 08:59:49.288712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.300517 (+ 11805us) service_pool.cc:225] Handling call
1215 08:59:51.086174 (+1785657us) spinlock_profiling.cc:243] Waited 65.5 ms on lock 0x7b540064d7d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:51.086344 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":150577920}
W20251215 08:59:51.088184 24080 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20251215 08:59:51.089411 21973 raft_consensus.cc:1275] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:51.091776 21973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 688) took 1071 ms. Trace:
I20251215 08:59:51.093459 21406 raft_consensus.cc:1240] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251215 08:59:51.093483 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.097671 21259 raft_consensus.cc:1275] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251215 08:59:51.100165 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.101545 21336 raft_consensus.cc:3060] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:51.105633 21413 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:51.055680 24364 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:51.051117 24208 raft_consensus.cc:697] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:51.107925 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.116729 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:51.111460 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:51.114087 21946 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:51.092124 21973 rpcz_store.cc:276] 1215 08:59:50.020485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:50.020611 (+ 126us) service_pool.cc:225] Handling call
1215 08:59:51.089054 (+1068443us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:51.089817 (+ 763us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:51.091582 (+ 1765us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540064d7d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:51.091715 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2374012160}
I20251215 08:59:51.060271 23984 raft_consensus.cc:697] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:51.122702 21332 raft_consensus.cc:1275] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:51.092211 24389 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251215 08:59:51.092048 24310 raft_consensus.cc:3055] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:51.125247 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.127391 21409 raft_consensus.cc:2468] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:51.073894 24346 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:51.128160 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.128975 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.132932 21337 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:51.133814 21406 raft_consensus.cc:1275] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer 0a2fcb89589c401d8efc8d86da4ba0f4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:51.134358 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:51.135360 24370 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:51.135247 23984 consensus_queue.cc:237] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.136878 24404 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251215 08:59:51.137089 24396 consensus_queue.cc:1048] T c6fb62879dd443a18eb94c506c7a82a2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 08:59:51.137603 24369 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:51.141659 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:51.068653 24363 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:51.144306 24363 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:51.145056 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:51.145754 21262 raft_consensus.cc:2393] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:51.068269 24335 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.135815 24370 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:51.146808 24370 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:51.147215 24301 log.cc:927] Time spent T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 1.786s user 0.000s sys 0.008s
I20251215 08:59:51.147640 24335 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:51.053521 24064 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:51.149286 24251 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:51.150408 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:51.151281 24251 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:51.151623 21341 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:51.124284 24229 consensus_queue.cc:1059] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20251215 08:59:51.152570 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:51.152972 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.153159 21259 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:51.154104 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.154699 21337 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:51.155512 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.156351 24245 consensus_queue.cc:1059] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:51.157070 24245 consensus_queue.cc:1243] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer 195a56ac72a84018880bfb3bf6e6ef1d log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251215 08:59:51.157182 24244 consensus_queue.cc:1059] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:51.158228 21339 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:51.128088 24310 raft_consensus.cc:740] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:51.159499 24310 consensus_queue.cc:260] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:51.160240 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.151346 24064 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:51.156528 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.157817 21232 leader_election.cc:304] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:51.162184 21337 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:51.167666 21413 raft_consensus.cc:1240] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:51.133313 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:51.133428 21332 raft_consensus.cc:2468] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:51.117213 24364 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:51.179431 21414 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:51.176913 24346 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.233736 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:51.202759 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.184249 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:51.239683 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 71657.7us
I20251215 08:59:51.189870 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.242875 21265 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251215 08:59:51.242336 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:51.233278 21336 raft_consensus.cc:2468] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:51.245920 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:51.240626 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 82877us
I20251215 08:59:51.246416 21336 raft_consensus.cc:2468] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:51.246634 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:51.247979 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.248979 24268 raft_consensus.cc:3055] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:51.249264 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.249954 21336 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:51.250849 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.251876 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:51.244709 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 111468us
I20251215 08:59:51.252857 21946 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:51.247288 21265 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:51.253710 21946 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:51.254459 24380 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:51.075398 24247 consensus_queue.cc:237] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.249413 24268 raft_consensus.cc:740] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:51.129868 24312 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:51.178622 24123 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:51.244400 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.259505 21337 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:51.247114 21232 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:51.260864 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:51.278350 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:51.307914 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.308660 21337 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:51.267930 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.309545 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:51.270764 21259 raft_consensus.cc:3055] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:51.310259 21341 raft_consensus.cc:2468] T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:51.309687 21336 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:51.311976 21379 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:51.312419 21263 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:51.269872 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:51.307469 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.272353 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46158 (request call id 676) took 2014 ms. Trace:
I20251215 08:59:51.314116 21978 rpcz_store.cc:276] 1215 08:59:49.257701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.260322 (+ 2621us) service_pool.cc:225] Handling call
1215 08:59:51.272090 (+2011768us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54004b19d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:51.272238 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2237998976}
I20251215 08:59:51.317821 24324 raft_consensus.cc:2804] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:51.318300 24324 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:51.318665 24324 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:51.321630 24268 consensus_queue.cc:260] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.323113 24346 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:51.326359 24369 raft_consensus.cc:2804] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:51.327003 24369 raft_consensus.cc:697] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:51.328038 24369 consensus_queue.cc:237] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:51.323154 24352 log.cc:927] Time spent T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20251215 08:59:51.330442 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:51.477591 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.270227 23974 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
W20251215 08:59:51.339629 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.339720 24069 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:51.255098 24251 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:51.283532 24372 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:51.215956 24229 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:51.481837 24380 raft_consensus.cc:697] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:51.255291 24396 consensus_queue.cc:1048] T bfcf2c8cd22a488aa741f1cede6e86ef P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.237s
W20251215 08:59:51.555424 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.477792 24335 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:51.627218 24335 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.271909 24336 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.596931 21332 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6.
I20251215 08:59:51.310446 21259 raft_consensus.cc:740] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:51.320662 24370 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:51.268661 24208 consensus_queue.cc:237] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.595177 24270 raft_consensus.cc:2749] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:51.500625 24069 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:51.718561 24405 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20251215 08:59:51.547238 24251 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:51.764372 24406 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20251215 08:59:51.866822 24196 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:51.871879 24336 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:51.671895 21259 consensus_queue.cc:260] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:51.665349 24335 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:51.955729 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.923373 24312 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:51.982494 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:52.132072 24161 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.977s user 0.015s sys 0.035s
W20251215 08:59:52.139428 21946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.020s sys 0.027s
W20251215 08:59:52.150097 21946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.839s user 0.020s sys 0.027s
W20251215 08:59:52.150976 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:51.949330 21259 raft_consensus.cc:3060] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:52.068120 21380 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.014s sys 0.023s
W20251215 08:59:52.194977 21380 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.014s sys 0.023s
W20251215 08:59:52.197546 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 44960.2us
W20251215 08:59:52.149722 24161 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.995s user 0.016s sys 0.035s
I20251215 08:59:51.828400 24380 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:52.189615 24364 consensus_meta.cc:287] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.931s user 0.007s sys 0.003s
I20251215 08:59:52.271111 21408 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:52.591387 21378 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.096s user 0.026s sys 0.051s
W20251215 08:59:52.595284 21378 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.100s user 0.027s sys 0.051s
W20251215 08:59:52.590153 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:52.593240 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 1.34611e+06us
W20251215 08:59:52.434576 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:52.594774 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:52.597546 21230 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.940s (SENT) [suppressed 40 similar messages]
I20251215 08:59:52.595340 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.600337 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.603484 21332 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
W20251215 08:59:52.322611 24324 consensus_meta.cc:287] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 1.004s user 0.013s sys 0.000s
W20251215 08:59:52.594810 24208 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.005s sys 0.006s
W20251215 08:59:52.593501 21944 connection.cc:232] RPC call timeout handler was delayed by -1.0335463189999246s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20251215 08:59:52.617033 21944 connection.cc:232] RPC call timeout handler was delayed by -1.0272812460000296s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18560000000000001s for a response
I20251215 08:59:52.608259 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1889) took 1572 ms. Trace:
I20251215 08:59:52.599557 21337 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
W20251215 08:59:52.618916 21337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1890) took 1575 ms (client timeout 2088 ms). Trace:
W20251215 08:59:52.619393 21337 rpcz_store.cc:269] 1215 08:59:51.043356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.309308 (+265952us) service_pool.cc:225] Handling call
1215 08:59:52.618842 (+1309534us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.607949 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:52.621136 21341 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:52.594316 24268 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:52.624020 21259 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251215 08:59:52.624549 21259 pending_rounds.cc:85] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 3
I20251215 08:59:52.624856 21259 pending_rounds.cc:107] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251215 08:59:52.625162 21259 raft_consensus.cc:2889] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:52.281718 24251 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:52.162523 24312 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.621659 21332 rpcz_store.cc:276] 1215 08:59:51.035442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.600019 (+1564577us) service_pool.cc:225] Handling call
1215 08:59:52.608185 (+ 8166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.626820 21259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2177) took 1358 ms. Trace:
I20251215 08:59:52.549957 24370 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:52.629245 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:52.625142 21341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 345) took 1545 ms (client timeout 2058 ms). Trace:
W20251215 08:59:52.650300 21341 rpcz_store.cc:269] 1215 08:59:51.079727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.601095 (+1521368us) service_pool.cc:225] Handling call
1215 08:59:52.625062 (+ 23967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.651675 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.653053 21332 raft_consensus.cc:2376] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6. Re-sending same reply.
I20251215 08:59:52.653844 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 350) took 1572 ms. Trace:
I20251215 08:59:52.654353 21332 rpcz_store.cc:276] 1215 08:59:51.081251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.651339 (+1570088us) service_pool.cc:225] Handling call
1215 08:59:52.653772 (+ 2433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.655031 21259 rpcz_store.cc:276] 1215 08:59:51.268155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.270373 (+ 2218us) service_pool.cc:225] Handling call
1215 08:59:51.270657 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.626259 (+1355602us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.626532 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.658456 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.659636 21337 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:52.660282 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 346) took 1580 ms. Trace:
I20251215 08:59:52.660724 21337 rpcz_store.cc:276] 1215 08:59:51.080016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.620329 (+1540313us) service_pool.cc:225] Handling call
1215 08:59:52.660187 (+ 39858us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:52.626089 24208 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s user 0.006s sys 0.006s
I20251215 08:59:52.663100 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:52.664641 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.671620 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.672633 21332 raft_consensus.cc:2468] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
I20251215 08:59:52.673311 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1878) took 1662 ms. Trace:
I20251215 08:59:52.673753 21332 rpcz_store.cc:276] 1215 08:59:51.010649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.671230 (+1660581us) service_pool.cc:225] Handling call
1215 08:59:52.673241 (+ 2011us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.674937 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f14cfff8c5a44ad78299ada0ca53baa9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.677907 21332 raft_consensus.cc:2393] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 9: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:52.282135 24364 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:52.653141 24303 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20251215 08:59:52.627039 24069 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:52.682725 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1893) took 1615 ms. Trace:
W20251215 08:59:52.681169 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 279709us
I20251215 08:59:52.698022 21341 rpcz_store.cc:276] 1215 08:59:51.067145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.664340 (+1597195us) service_pool.cc:225] Handling call
1215 08:59:52.682632 (+ 18292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.699502 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:52.700047 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.700147 21979 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 8
I20251215 08:59:52.700788 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1881) took 1689 ms. Trace:
I20251215 08:59:52.701176 21341 rpcz_store.cc:276] 1215 08:59:51.011597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.699801 (+1688204us) service_pool.cc:225] Handling call
1215 08:59:52.700730 (+ 929us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.702073 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.702982 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1894) took 1635 ms. Trace:
I20251215 08:59:52.703361 21341 rpcz_store.cc:276] 1215 08:59:51.067391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.701746 (+1634355us) service_pool.cc:225] Handling call
1215 08:59:52.702905 (+ 1159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.704380 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:52.663096 24369 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.003s sys 0.025s
I20251215 08:59:52.662665 24304 raft_consensus.cc:493] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:52.706408 21341 raft_consensus.cc:1813] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Handling vote request from an unknown peer beefe15185ac42ef9a20251007f39a56
I20251215 08:59:52.733294 21341 raft_consensus.cc:2410] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
W20251215 08:59:52.707638 24369 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s user 0.004s sys 0.025s
I20251215 08:59:52.708503 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 348) took 1627 ms. Trace:
I20251215 08:59:52.733855 21332 rpcz_store.cc:276] 1215 08:59:51.080590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.674631 (+1594041us) service_pool.cc:225] Handling call
1215 08:59:52.708184 (+ 33553us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.734786 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.735538 21332 raft_consensus.cc:2410] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:52.700801 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.735975 21337 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:52.736411 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1952) took 1577 ms. Trace:
I20251215 08:59:52.736670 21337 rpcz_store.cc:276] 1215 08:59:51.158651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.661624 (+1502973us) service_pool.cc:225] Handling call
1215 08:59:52.736359 (+ 74735us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:52.709396 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:52.687409 24370 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:52.728638 21307 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:52.728776 21262 raft_consensus.cc:1240] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:52.750485 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1892) took 1683 ms. Trace:
I20251215 08:59:52.750934 21332 rpcz_store.cc:276] 1215 08:59:51.066820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.734560 (+1667740us) service_pool.cc:225] Handling call
1215 08:59:52.750425 (+ 15865us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.753768 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1874) took 1745 ms. Trace:
I20251215 08:59:52.754179 21341 rpcz_store.cc:276] 1215 08:59:51.008221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.704106 (+1695885us) service_pool.cc:225] Handling call
1215 08:59:52.753699 (+ 49593us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.755802 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:52.731097 24372 consensus_meta.cc:287] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.184s user 0.011s sys 0.003s
I20251215 08:59:52.629664 24324 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:52.731137 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:52.707528 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 110018us
W20251215 08:59:52.684460 24344 log.cc:927] Time spent T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251215 08:59:52.738722 21979 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 8.
I20251215 08:59:52.737579 24251 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:52.785353 21978 raft_consensus.cc:1240] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [term 7 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:52.729954 24409 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:52.851795 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:52.684276 24419 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20251215 08:59:52.854102 21332 raft_consensus.cc:1240] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:52.854848 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1873) took 1846 ms. Trace:
I20251215 08:59:52.855257 21332 rpcz_store.cc:276] 1215 08:59:51.008001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.761614 (+1753613us) service_pool.cc:225] Handling call
1215 08:59:52.853199 (+ 91585us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.854584 (+ 1385us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.854791 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.856246 21341 raft_consensus.cc:1275] T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:52.663455 24362 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:52.726579 24274 raft_consensus.cc:3055] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
W20251215 08:59:52.852692 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:52.852998 21337 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:52.809356 24422 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:52.857033 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1875) took 1848 ms. Trace:
I20251215 08:59:52.861658 21341 rpcz_store.cc:276] 1215 08:59:51.008475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.762998 (+1754523us) service_pool.cc:225] Handling call
1215 08:59:52.855702 (+ 92704us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.856729 (+ 1027us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.856959 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.860277 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1871) took 1852 ms. Trace:
I20251215 08:59:52.862744 21337 rpcz_store.cc:276] 1215 08:59:51.007544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.757353 (+1749809us) service_pool.cc:225] Handling call
1215 08:59:52.860184 (+102831us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.763799 24304 raft_consensus.cc:515] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:52.859395 21307 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:52.864517 21341 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251215 08:59:52.868314 21406 raft_consensus.cc:1240] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:52.808353 24336 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:52.806497 24009 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:52.870292 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1886) took 1857 ms. Trace:
I20251215 08:59:52.871567 21341 rpcz_store.cc:276] 1215 08:59:51.013192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.863895 (+1850703us) service_pool.cc:225] Handling call
1215 08:59:52.864257 (+ 362us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:52.869940 (+ 5683us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.870206 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.851538 24369 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:52.876458 21332 raft_consensus.cc:1275] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20251215 08:59:52.887392 21259 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251215 08:59:52.890641 21332 pending_rounds.cc:85] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 3
I20251215 08:59:52.890808 21259 pending_rounds.cc:85] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 3
I20251215 08:59:52.891007 21332 pending_rounds.cc:107] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20251215 08:59:52.891355 21259 pending_rounds.cc:107] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251215 08:59:52.891515 21332 raft_consensus.cc:2889] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:52.763382 24370 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.892107 21332 pending_rounds.cc:107] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20251215 08:59:52.896215 21332 raft_consensus.cc:2889] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:52.889923 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1940) took 1869 ms. Trace:
I20251215 08:59:52.891992 21259 raft_consensus.cc:2889] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:52.861231 24274 raft_consensus.cc:740] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:52.861076 24372 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:52.873587 24337 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.840s user 0.063s sys 0.106s
W20251215 08:59:52.886279 24412 log.cc:927] Time spent T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20251215 08:59:52.897619 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:52.896519 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 292435us
I20251215 08:59:52.858789 24425 raft_consensus.cc:2749] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:52.908174 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1883) took 1896 ms. Trace:
I20251215 08:59:52.897267 21341 rpcz_store.cc:276] 1215 08:59:51.019973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.872439 (+1852466us) service_pool.cc:225] Handling call
1215 08:59:52.872739 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.872914 (+ 175us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:52.872941 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:52.872994 (+ 53us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:52.873075 (+ 81us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.889362 (+ 16287us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:52.889684 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.874271 24208 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:52.911940 21332 rpcz_store.cc:276] 1215 08:59:51.012238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.867030 (+1854792us) service_pool.cc:225] Handling call
1215 08:59:52.868919 (+ 1889us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.907896 (+ 38977us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.908102 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:52.911257 21265 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:52.912926 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 284421us
W20251215 08:59:52.909550 24337 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.877s user 0.065s sys 0.107s
I20251215 08:59:52.874910 24420 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:52.916114 21341 raft_consensus.cc:1240] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:52.858889 24251 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:52.923729 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:52.923640 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1888) took 1893 ms. Trace:
I20251215 08:59:52.924557 21259 raft_consensus.cc:2410] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20251215 08:59:52.924432 21341 rpcz_store.cc:276] 1215 08:59:51.030210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.915489 (+1885279us) service_pool.cc:225] Handling call
1215 08:59:52.915875 (+ 386us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.923411 (+ 7536us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.923589 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:52.921397 21944 leader_election.cc:336] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.711s (SENT)
I20251215 08:59:52.886454 24336 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.910894 24431 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:52.899519 24426 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:52.913558 24144 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:52.894315 24423 consensus_queue.cc:1059] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20251215 08:59:52.936781 21337 raft_consensus.cc:1240] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6. Ops: []
I20251215 08:59:52.938652 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1910) took 4846 ms. Trace:
I20251215 08:59:52.944084 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1931) took 3002 ms. Trace:
I20251215 08:59:52.961810 21333 rpcz_store.cc:276] 1215 08:59:48.092456 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:48.140603 (+ 48147us) service_pool.cc:225] Handling call
1215 08:59:48.173188 (+ 32585us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:49.266745 (+1093557us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.938351 (+3671606us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400153e54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:52.938556 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2513289472}
I20251215 08:59:52.932986 21265 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:52.861837 24409 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.935820 24431 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.926564 24429 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:52.975234 24429 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.976194 24409 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:52.976310 24431 leader_election.cc:290] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:52.867538 24422 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.928524 21257 raft_consensus.cc:1275] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:52.976900 24425 raft_consensus.cc:493] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:52.937812 24408 log.cc:927] Time spent T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20251215 08:59:52.977496 24425 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:52.945261 24415 log.cc:927] Time spent T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20251215 08:59:52.963670 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:52.938105 21415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1841) took 3304 ms (client timeout 1660 ms). Trace:
I20251215 08:59:52.863977 24362 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.963907 21333 raft_consensus.cc:1240] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:52.979264 21336 raft_consensus.cc:2410] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251215 08:59:52.979012 21415 rpcz_store.cc:269] 1215 08:59:49.633375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.710897 (+ 77522us) service_pool.cc:225] Handling call
1215 08:59:52.937855 (+3226958us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54000d11d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:52.938013 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":558667136}
I20251215 08:59:52.940339 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1571) took 1791 ms. Trace:
I20251215 08:59:52.980465 21262 rpcz_store.cc:276] 1215 08:59:51.148866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.149642 (+ 776us) service_pool.cc:225] Handling call
1215 08:59:51.149905 (+ 263us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.747671 (+1597766us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.940048 (+192377us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400170554. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:52.940250 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3629813504}
I20251215 08:59:52.955246 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:52.927587 21944 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:52.982678 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1864) took 1922 ms. Trace:
W20251215 08:59:52.982940 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 61574.5us
I20251215 08:59:52.983161 21410 rpcz_store.cc:276] 1215 08:59:51.059712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.078893 (+ 19181us) service_pool.cc:225] Handling call
1215 08:59:52.955074 (+1876181us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b5800264c98. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8e3e 000000000035a3c9 00007f398caf3560 00007f398cbbbdd4 00007f398cbbbe41 00007f398cb69bd0 00007f398cb694b4 00007f39860f9834 00007f39860f97c2
1215 08:59:52.982602 (+ 27528us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33702272}
I20251215 08:59:52.886765 24009 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:52.935019 24251 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:52.974399 24414 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:52.921947 24208 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:52.980055 21336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1944) took 1947 ms (client timeout 2154 ms). Trace:
I20251215 08:59:52.936187 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:52.992748 21335 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:52.993325 21335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 353) took 1911 ms (client timeout 2168 ms). Trace:
W20251215 08:59:52.944507 24418 log.cc:927] Time spent T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20251215 08:59:52.962139 21334 rpcz_store.cc:276] 1215 08:59:49.941380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:50.066381 (+125001us) service_pool.cc:225] Handling call
1215 08:59:50.066717 (+ 336us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:50.067031 (+ 314us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:50.067066 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:50.067559 (+ 493us) log.cc:844] Serialized 26 byte log entry
1215 08:59:52.741387 (+2673828us) spinlock_profiling.cc:243] Waited 875 us on lock 0x7b180046c420. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ae8977 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b057f2 00007f398774d5ab 00007f39877530c3 00007f3987753471 00007f3987b6f73f
1215 08:59:52.857698 (+116311us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:52.857753 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.857857 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:52.943779 (+ 85922us) raft_consensus.cc:1685] finished
1215 08:59:52.943802 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:52.944003 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8502,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":11449,"spinlock_wait_cycles":2012032,"thread_start_us":107834,"threads_started":1,"wal-append.queue_time_us":2805692}
I20251215 08:59:52.928581 24433 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:52.899143 24369 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.910490 24372 raft_consensus.cc:515] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:52.943161 24411 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20251215 08:59:52.978384 24422 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:52.994628 21336 rpcz_store.cc:269] 1215 08:59:51.032943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.936826 (+1903883us) service_pool.cc:225] Handling call
1215 08:59:52.979983 (+ 43157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58624}
W20251215 08:59:52.984433 21944 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.927s (SENT)
I20251215 08:59:52.986037 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1897) took 1917 ms. Trace:
W20251215 08:59:53.000535 21230 connection.cc:232] RPC call timeout handler was delayed by -1.3820994600000631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.326s for a response
I20251215 08:59:53.024891 21333 rpcz_store.cc:276] 1215 08:59:51.068194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.962995 (+1894801us) service_pool.cc:225] Handling call
1215 08:59:52.963728 (+ 733us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.979504 (+ 15776us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:52.985962 (+ 6458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52096}
W20251215 08:59:53.001883 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 138572us
I20251215 08:59:52.955262 24364 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:52.959383 24370 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:52.993765 21335 rpcz_store.cc:269] 1215 08:59:51.082075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.935905 (+1853830us) service_pool.cc:225] Handling call
1215 08:59:52.993247 (+ 57342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:53.029033 21335 raft_consensus.cc:1275] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:52.947088 24324 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:53.020748 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1948) took 1947 ms. Trace:
I20251215 08:59:53.029896 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1943) took 1999 ms. Trace:
I20251215 08:59:52.989912 24409 raft_consensus.cc:2749] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:53.029896 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1949) took 1918 ms. Trace:
I20251215 08:59:53.030295 21334 rpcz_store.cc:276] 1215 08:59:51.073706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.997997 (+1924291us) service_pool.cc:225] Handling call
1215 08:59:53.020019 (+ 22022us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.020284 (+ 265us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.020323 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:53.020387 (+ 64us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:53.020437 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.020514 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.020666 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:53.031898 21335 rpcz_store.cc:276] 1215 08:59:51.111634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.028216 (+1916582us) service_pool.cc:225] Handling call
1215 08:59:53.028555 (+ 339us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.029692 (+ 1137us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.029841 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:53.030372 21341 rpcz_store.cc:276] 1215 08:59:51.030562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.925376 (+1894814us) service_pool.cc:225] Handling call
1215 08:59:52.925690 (+ 314us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:52.961905 (+ 36215us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:52.961939 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:52.985938 (+ 23999us) log.cc:844] Serialized 26 byte log entry
1215 08:59:53.025569 (+ 39631us) raft_consensus.cc:1645] Marking committed up to 1
1215 08:59:53.025653 (+ 84us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.025756 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:53.029687 (+ 3931us) raft_consensus.cc:1685] finished
1215 08:59:53.029711 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.029835 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3584,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":119,"thread_start_us":14450,"threads_started":1,"wal-append.queue_time_us":32097}
W20251215 08:59:53.031205 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 46809.6us
I20251215 08:59:53.034958 21341 raft_consensus.cc:1275] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:53.035530 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1953) took 1876 ms. Trace:
W20251215 08:59:53.035777 21944 leader_election.cc:336] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.207s (SENT)
W20251215 08:59:53.036652 21944 leader_election.cc:336] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.760s (SENT)
I20251215 08:59:53.037038 21944 leader_election.cc:304] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:53.035810 21341 rpcz_store.cc:276] 1215 08:59:51.158952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.034382 (+1875430us) service_pool.cc:225] Handling call
1215 08:59:53.034642 (+ 260us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.035296 (+ 654us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.035481 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:53.039984 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1954) took 1880 ms. Trace:
I20251215 08:59:53.040400 21341 rpcz_store.cc:276] 1215 08:59:51.159294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.038948 (+1879654us) service_pool.cc:225] Handling call
1215 08:59:53.039337 (+ 389us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.039553 (+ 216us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.039589 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:53.039648 (+ 59us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:53.039695 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.039765 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.039905 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:53.042515 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1955) took 1882 ms. Trace:
I20251215 08:59:53.042919 21341 rpcz_store.cc:276] 1215 08:59:51.159627 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.041525 (+1881898us) service_pool.cc:225] Handling call
1215 08:59:53.041875 (+ 350us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.042095 (+ 220us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.042132 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:53.042192 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:53.042236 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.042309 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.042450 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:53.056524 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:53.057096 21230 connection.cc:232] RPC call timeout handler was delayed by -1.2951478360000692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2565s for a response
I20251215 08:59:52.996526 24427 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:52.997238 24312 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.006425 24431 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:52.991672 24144 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.066015 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1957) took 1788 ms. Trace:
I20251215 08:59:53.080780 21341 rpcz_store.cc:276] 1215 08:59:51.257031 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.044008 (+1786977us) service_pool.cc:225] Handling call
1215 08:59:53.044365 (+ 357us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.044583 (+ 218us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.044624 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:53.044685 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:53.044729 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.044800 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.044936 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:53.074021 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 161913us
I20251215 08:59:52.926872 24420 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.933956 24274 consensus_queue.cc:260] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:52.996280 24414 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.067236 24429 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:53.097630 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.093091 21378 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:53.105356 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.014024 24433 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:53.082046 24312 raft_consensus.cc:2749] T f14cfff8c5a44ad78299ada0ca53baa9 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:53.081267 24427 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:53.070405 24251 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:53.173902 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 80918.3us
I20251215 08:59:53.135476 24423 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:53.135083 24304 leader_election.cc:290] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.156482 24440 raft_consensus.cc:3055] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
W20251215 08:59:53.195097 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 167586us
I20251215 08:59:53.176019 24429 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:53.198029 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:53.176443 24434 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:53.176559 24380 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:53.169695 23846 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:53.086378 24431 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.198711 21979 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 1.
I20251215 08:59:53.201040 21378 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:53.206058 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 346722us
I20251215 08:59:53.207906 24434 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.194286 23959 raft_consensus.cc:2804] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:53.215401 21261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2148) took 3360 ms. Trace:
I20251215 08:59:53.220634 21307 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:53.195554 24009 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:53.222106 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 354) took 2139 ms. Trace:
W20251215 08:59:53.267753 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.258409 21261 rpcz_store.cc:276] 1215 08:59:49.855283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.855454 (+ 171us) service_pool.cc:225] Handling call
1215 08:59:49.893184 (+ 37730us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:50.056743 (+163559us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:50.056772 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:50.056814 (+ 42us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:50.056846 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.215158 (+3158312us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b5400323a54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:53.215206 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.215338 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":375724288}
W20251215 08:59:53.268213 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:53.270525 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 49995.5us
I20251215 08:59:53.271297 21307 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:53.266739 23959 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:53.195272 24251 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:53.196059 24440 raft_consensus.cc:740] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:53.289499 21307 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:53.214389 24144 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:53.304910 21307 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:53.306967 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 106081us
I20251215 08:59:53.222437 24380 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:53.273463 24438 log.cc:927] Time spent T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20251215 08:59:53.222654 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:53.277953 24445 log.cc:927] Time spent T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
W20251215 08:59:53.307045 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 395641us
I20251215 08:59:53.342082 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:53.281349 24236 raft_consensus.cc:3060] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:53.278187 24304 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:53.258834 24426 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:53.208460 24454 raft_consensus.cc:493] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:53.223481 24268 consensus_meta.cc:287] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.592s user 0.010s sys 0.003s
I20251215 08:59:53.316555 24425 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:53.325351 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 446157us
I20251215 08:59:53.272580 23959 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:53.274802 24251 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:53.274956 24208 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.271641 24452 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:53.276578 24427 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:53.404784 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:53.364959 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47830 (request call id 355) took 2282 ms. Trace:
I20251215 08:59:53.245337 23846 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.195742 24423 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:53.279520 24420 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.273254 24414 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:53.204721 24429 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.278628 24433 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:53.391834 24465 raft_consensus.cc:493] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:53.296614 24364 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:53.360020 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.363348 24453 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:53.308029 24258 raft_consensus.cc:3055] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:53.282035 24434 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:53.387809 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:53.387437 21413 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:53.392328 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:53.398254 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.398181 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:53.396708 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.400938 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:53.402379 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:53.405897 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1942) took 2375 ms. Trace:
I20251215 08:59:53.308661 24144 consensus_queue.cc:1048] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20251215 08:59:53.389741 24372 leader_election.cc:290] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.408119 21971 raft_consensus.cc:2410] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:53.408488 21336 rpcz_store.cc:276] 1215 08:59:51.082570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.026106 (+1943536us) service_pool.cc:225] Handling call
1215 08:59:53.026459 (+ 353us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:53.027252 (+ 793us) raft_consensus.cc:1534] Early marking committed up to index 3
1215 08:59:53.027285 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:53.030400 (+ 3115us) log.cc:844] Serialized 26 byte log entry
1215 08:59:53.189747 (+159347us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:53.189807 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.189932 (+ 125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:53.364679 (+174747us) raft_consensus.cc:1685] finished
1215 08:59:53.364705 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.364890 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":135,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":117,"thread_start_us":158943,"threads_started":1,"wal-append.queue_time_us":251535}
I20251215 08:59:53.366693 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:53.395962 24451 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:53.438614 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 137340us
I20251215 08:59:53.405589 24362 leader_election.cc:290] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:53.409538 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:53.407199 24464 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:53.393184 24454 raft_consensus.cc:515] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.416543 24427 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:53.535346 21260 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 2.
I20251215 08:59:53.387029 24461 raft_consensus.cc:2749] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:53.403857 24426 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:53.385550 24440 consensus_queue.cc:260] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:53.412542 21230 leader_election.cc:336] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.679s (SENT)
I20251215 08:59:53.543109 21307 leader_election.cc:304] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:53.656147 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.403472 24304 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:53.267805 21333 rpcz_store.cc:276] 1215 08:59:51.082329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.025838 (+1943509us) service_pool.cc:225] Handling call
1215 08:59:53.026202 (+ 364us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.026420 (+ 218us) raft_consensus.cc:1534] Early marking committed up to index 7
1215 08:59:53.168909 (+142489us) log.cc:844] Serialized 14 byte log entry
1215 08:59:53.221766 (+ 52857us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:53.221820 (+ 54us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:53.221863 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.221930 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.222052 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":181449,"raft.queue_time_us":83,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":44426,"thread_start_us":194517,"threads_started":2,"wal-append.queue_time_us":111836}
W20251215 08:59:53.778399 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.900943 21257 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
W20251215 08:59:53.901346 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 488809us
W20251215 08:59:53.558646 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 160725us
I20251215 08:59:53.557392 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:53.453109 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:53.534204 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:53.570713 24323 log.cc:927] Time spent T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 3.685s user 0.018s sys 0.014s
W20251215 08:59:53.678061 24470 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.222s user 0.001s sys 0.000s
I20251215 08:59:53.418515 24369 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.951946 21409 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 10.
W20251215 08:59:53.826889 24407 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 2.182s user 0.003s sys 0.000s
I20251215 08:59:53.542563 21332 rpcz_store.cc:276] 1215 08:59:51.030367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.917580 (+1887213us) service_pool.cc:225] Handling call
1215 08:59:52.917995 (+ 415us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:52.918271 (+ 276us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:52.918310 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:52.924352 (+ 6042us) log.cc:844] Serialized 26 byte log entry
1215 08:59:53.058192 (+133840us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:53.058237 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.058310 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:53.405664 (+347354us) raft_consensus.cc:1685] finished
1215 08:59:53.405684 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.405815 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":62393,"threads_started":1,"wal-append.queue_time_us":105269}
W20251215 08:59:53.895635 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 352638us
I20251215 08:59:53.912058 21410 raft_consensus.cc:1275] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20251215 08:59:53.953649 21230 leader_election.cc:336] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.944s (SENT)
I20251215 08:59:53.902870 21407 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
W20251215 08:59:53.960919 21230 leader_election.cc:336] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.088s (SENT)
I20251215 08:59:53.475730 24258 raft_consensus.cc:740] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:53.891848 24009 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.000s sys 0.009s
I20251215 08:59:53.595055 24431 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.488826 24380 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.674470 24349 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:53.483067 24310 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:53.900341 24426 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:53.482721 24429 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:53.961710 21307 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:53.962178 21265 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.000s sys 0.016s
W20251215 08:59:53.969265 21265 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.000s sys 0.016s
I20251215 08:59:53.484669 23846 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:53.475319 24453 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:53.899286 24469 raft_consensus.cc:2749] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251215 08:59:53.971375 21946 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.974069 21307 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:53.596935 23959 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:53.962571 24009 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.626s user 0.003s sys 0.012s
I20251215 08:59:53.994714 21411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 306) took 4526 ms. Trace:
I20251215 08:59:53.961329 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:53.475919 24465 raft_consensus.cc:515] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.545535 24364 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:53.446367 24433 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.533s
I20251215 08:59:53.913807 24427 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:53.545382 24451 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.000494 21946 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:54.010834 21946 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:54.005477 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:54.012105 21409 raft_consensus.cc:2393] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 5: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:53.980185 24304 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251215 08:59:53.995239 21411 rpcz_store.cc:276] 1215 08:59:49.456298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.456474 (+ 176us) service_pool.cc:225] Handling call
1215 08:59:49.462664 (+ 6190us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:49.474479 (+ 11815us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:49.474511 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:49.482091 (+ 7580us) log.cc:844] Serialized 52 byte log entry
1215 08:59:49.699779 (+217688us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:49.699821 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:49.699920 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:53.360637 (+3660717us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:53.977743 (+617106us) raft_consensus.cc:1685] finished
1215 08:59:53.977764 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:53.982880 (+ 5116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3083,"raft.queue_time_us":6939,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":9450,"spinlock_wait_cycles":20468608,"thread_start_us":217181,"threads_started":1,"wal-append.queue_time_us":385716}
I20251215 08:59:54.008420 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.009147 21415 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:54.017165 21407 raft_consensus.cc:2393] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:53.968319 24426 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:53.409605 24452 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:54.010792 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:53.557221 24464 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:53.968783 24349 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.023836 24369 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251215 08:59:54.027944 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 70665.1us
I20251215 08:59:54.029610 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.030649 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.009248 24469 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:54.018050 24085 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20251215 08:59:54.025902 24236 consensus_meta.cc:287] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.667s user 0.007s sys 0.004s
I20251215 08:59:54.060591 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:54.061396 21407 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:54.058528 24085 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:54.063719 24085 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Advancing to term 11
I20251215 08:59:54.063908 21977 raft_consensus.cc:1275] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20251215 08:59:54.064327 21977 pending_rounds.cc:85] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Aborting all ops after (but not including) 8
I20251215 08:59:54.064553 21977 pending_rounds.cc:107] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20251215 08:59:54.064754 21977 raft_consensus.cc:2889] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:54.065380 24343 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.065860 24343 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.008587 24380 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
W20251215 08:59:54.071312 24477 log.cc:927] Time spent T 79790fd760ff4ae1b3879326decc531a P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20251215 08:59:54.072579 23959 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.058810 24369 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0a2fcb89589c401d8efc8d86da4ba0f4)
I20251215 08:59:54.073189 24369 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:54.074359 24453 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:53.982760 24258 consensus_queue.cc:260] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.076097 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1887) took 3062 ms. Trace:
I20251215 08:59:54.076749 24465 leader_election.cc:290] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.078388 24452 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.059073 21381 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:54.078920 21978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46154 (request call id 725) took 1353 ms. Trace:
I20251215 08:59:54.079262 21978 rpcz_store.cc:276] 1215 08:59:52.725404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.726925 (+ 1521us) service_pool.cc:225] Handling call
1215 08:59:52.727274 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.852162 (+124888us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.078682 (+1226520us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b54005fdf54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.078856 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80996096}
I20251215 08:59:54.079896 21381 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:54.076499 21337 rpcz_store.cc:276] 1215 08:59:51.013492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.866465 (+1852973us) service_pool.cc:225] Handling call
1215 08:59:52.889365 (+ 22900us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.961521 (+ 72156us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.075835 (+1114314us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b5400298454. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.076024 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108567296}
I20251215 08:59:54.081256 24381 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251215 08:59:54.081766 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.084255 21414 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:54.083770 21411 raft_consensus.cc:2393] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:54.076107 24258 raft_consensus.cc:3060] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:54.096154 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2159) took 3250 ms. Trace:
I20251215 08:59:54.095012 21409 raft_consensus.cc:1240] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251215 08:59:54.096698 21258 rpcz_store.cc:276] 1215 08:59:50.845900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:50.863472 (+ 17572us) service_pool.cc:225] Handling call
1215 08:59:50.863779 (+ 307us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:50.916689 (+ 52910us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.095854 (+3179165us) spinlock_profiling.cc:243] Waited 52.2 ms on lock 0x7b5400170a54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.096054 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120109952}
I20251215 08:59:54.092422 23851 raft_consensus.cc:2804] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:54.105401 21408 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1847) took 4362 ms. Trace:
I20251215 08:59:54.105571 24227 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.105809 21408 rpcz_store.cc:276] 1215 08:59:49.748229 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.756715 (+ 8486us) service_pool.cc:225] Handling call
1215 08:59:49.848533 (+ 91818us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.594575 (+2746042us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.105166 (+1510591us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b5400341f54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.105337 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5570703104}
I20251215 08:59:54.118515 21409 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:54.122292 21407 raft_consensus.cc:2468] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:54.085930 24009 consensus_queue.cc:1048] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20251215 08:59:54.080741 24481 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
W20251215 08:59:54.138725 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.153306 23851 raft_consensus.cc:697] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:54.154366 23851 consensus_queue.cc:237] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.157656 24364 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.157866 24299 raft_consensus.cc:3060] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:54.163173 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:54.163786 21336 raft_consensus.cc:3060] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:54.164062 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.164649 21332 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:54.165669 21333 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20251215 08:59:54.166695 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.167366 21333 raft_consensus.cc:2393] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:54.168123 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:54.168603 24455 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.890s user 0.001s sys 0.010s
I20251215 08:59:54.169826 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:54.170549 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.183930 21337 raft_consensus.cc:2393] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:54.170548 21336 raft_consensus.cc:2468] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:54.171489 21258 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20251215 08:59:54.184887 21258 pending_rounds.cc:85] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 4
I20251215 08:59:54.173491 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:54.175554 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.176148 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:54.177175 21339 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 1939) took 3166 ms (client timeout 1664 ms). Trace:
I20251215 08:59:54.178236 21412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1867) took 3117 ms. Trace:
W20251215 08:59:54.181519 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 220597us
I20251215 08:59:54.109397 24423 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.185164 21258 pending_rounds.cc:107] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20251215 08:59:54.191195 21258 raft_consensus.cc:2889] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:54.192610 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.129153 24454 leader_election.cc:290] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.205915 24427 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.108867 24227 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:54.188015 21339 rpcz_store.cc:269] 1215 08:59:51.011208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.031284 (+ 20076us) service_pool.cc:225] Handling call
1215 08:59:54.176897 (+3145613us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54002788d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:54.177066 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":267848448}
I20251215 08:59:54.188243 21412 rpcz_store.cc:276] 1215 08:59:51.060402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.078961 (+ 18559us) service_pool.cc:225] Handling call
1215 08:59:51.079224 (+ 263us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:51.184970 (+105746us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:51.184999 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:51.185068 (+ 69us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:51.185097 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.177927 (+2992830us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b540034fb54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.178004 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:54.178157 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242767232}
I20251215 08:59:54.187031 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.208416 21261 raft_consensus.cc:2410] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:54.197103 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.209690 24336 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.210142 21257 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:54.185106 24482 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:54.216554 24464 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:54.217381 21340 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1869) took 3213 ms (client timeout 2247 ms). Trace:
W20251215 08:59:54.217787 21340 rpcz_store.cc:269] 1215 08:59:51.003965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.004117 (+ 152us) service_pool.cc:225] Handling call
1215 08:59:54.217159 (+3213042us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54001252d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:54.217310 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3857139072}
I20251215 08:59:54.220589 21979 raft_consensus.cc:1275] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20251215 08:59:54.221027 21979 pending_rounds.cc:85] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Aborting all ops after (but not including) 7
I20251215 08:59:54.221254 21979 pending_rounds.cc:107] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Aborting uncommitted NO_OP operation due to leader change: 8.8
W20251215 08:59:54.189527 21230 leader_election.cc:336] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.247s (SENT)
I20251215 08:59:54.221488 21979 raft_consensus.cc:2889] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:54.221819 21338 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 LEADER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:54.221798 21230 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:54.198825 23959 consensus_queue.cc:1059] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
W20251215 08:59:54.199589 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.200075 24476 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:54.200610 24465 raft_consensus.cc:493] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.209329 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:54.225158 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 214424us
I20251215 08:59:54.225623 21258 raft_consensus.cc:2393] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 5: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:54.234692 21230 leader_election.cc:336] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.260s (SENT)
W20251215 08:59:54.236622 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 50884.1us
W20251215 08:59:54.237214 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:54.237538 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:54.239944 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:54.242798 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 268805us
I20251215 08:59:54.235085 21229 leader_election.cc:304] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:54.276134 21307 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:54.278656 21307 leader_election.cc:400] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7.
I20251215 08:59:54.279103 21307 leader_election.cc:403] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:54.225268 24141 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.278390 21946 leader_election.cc:304] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:54.283236 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.282486 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:54.284768 21381 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:54.288559 24236 raft_consensus.cc:2804] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 08:59:54.289058 24236 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:54.289196 24161 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:54.289556 24236 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 1 FOLLOWER]: Advancing to term 2
I20251215 08:59:54.294031 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:54.296121 24068 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251215 08:59:54.296468 24343 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.296744 24481 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251215 08:59:54.297169 24481 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:54.297544 24481 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:54.299479 24451 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.297173 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1951) took 3139 ms. Trace:
I20251215 08:59:54.301509 21335 rpcz_store.cc:276] 1215 08:59:51.158250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.032637 (+1874387us) service_pool.cc:225] Handling call
1215 08:59:53.032902 (+ 265us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:53.036483 (+ 3581us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.036511 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:53.037037 (+ 526us) log.cc:844] Serialized 26 byte log entry
1215 08:59:53.169388 (+132351us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:53.169435 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.169514 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:54.167936 (+998422us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:54.283363 (+115427us) raft_consensus.cc:1685] finished
1215 08:59:54.283387 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:54.297109 (+ 13722us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":116,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":12436864,"thread_start_us":131969,"threads_started":1,"wal-append.queue_time_us":231673}
I20251215 08:59:54.235682 24476 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.305910 24236 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.307839 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.308648 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.310292 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.311543 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:54.311791 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.309468 21333 raft_consensus.cc:2468] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:54.312844 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.313498 21339 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:54.314070 21333 raft_consensus.cc:3060] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:54.314467 21339 raft_consensus.cc:3055] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:54.314857 21339 raft_consensus.cc:740] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:54.315582 21339 consensus_queue.cc:260] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.317157 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:54.317984 21336 raft_consensus.cc:2410] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:54.321651 21338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1936) took 3317 ms. Trace:
I20251215 08:59:54.322046 21338 rpcz_store.cc:276] 1215 08:59:51.004648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.004796 (+ 148us) service_pool.cc:225] Handling call
1215 08:59:54.221560 (+3216764us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:54.222475 (+ 915us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.321431 (+ 98956us) spinlock_profiling.cc:243] Waited 3.19 s on lock 0x7b54001252d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:54.321574 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7336722688}
I20251215 08:59:54.324306 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.325040 24336 consensus_queue.cc:1059] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:54.326447 21259 raft_consensus.cc:3055] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:54.326591 24336 raft_consensus.cc:3055] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Stepping down as leader of term 9
I20251215 08:59:54.326835 21259 raft_consensus.cc:740] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:54.326977 24336 raft_consensus.cc:740] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:54.327735 24336 consensus_queue.cc:260] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.328351 21406 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 324) took 3071 ms. Trace:
I20251215 08:59:54.330588 24064 consensus_queue.cc:1048] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:54.279462 24469 consensus_queue.cc:1059] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20251215 08:59:54.330619 24215 raft_consensus.cc:2749] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447) with higher term. Message: Invalid argument: T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7.
I20251215 08:59:54.334355 24469 consensus_queue.cc:1048] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:54.334936 24215 raft_consensus.cc:3055] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:54.335397 24215 raft_consensus.cc:740] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:54.329627 23974 consensus_queue.cc:1048] T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20251215 08:59:54.328668 21406 rpcz_store.cc:276] 1215 08:59:51.257211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:51.257336 (+ 125us) service_pool.cc:225] Handling call
1215 08:59:51.355770 (+ 98434us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:52.868791 (+1513021us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.328145 (+1459354us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54001757d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.328295 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3477950208}
I20251215 08:59:54.234313 24475 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.236263 24427 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.327617 21259 consensus_queue.cc:260] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.348101 21259 raft_consensus.cc:3060] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:54.316469 21339 raft_consensus.cc:3060] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
W20251215 08:59:54.352203 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.233315 24482 raft_consensus.cc:2749] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:54.279637 24304 raft_consensus.cc:2749] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251215 08:59:54.281275 24439 raft_consensus.cc:2804] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 08:59:54.357494 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:54.328891 24336 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:54.359553 21340 raft_consensus.cc:2410] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:54.336369 24215 consensus_queue.cc:260] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.360510 21379 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:54.344952 24343 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:54.366362 21379 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:54.382697 21333 raft_consensus.cc:1275] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:54.242108 24465 raft_consensus.cc:515] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.395406 21339 raft_consensus.cc:1275] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20251215 08:59:54.432680 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 49162.1us
W20251215 08:59:54.393088 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.488225 21339 pending_rounds.cc:85] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:54.443658 24227 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.382697 24343 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:54.363301 24439 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:54.409289 23974 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:54.444204 24349 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.444365 24409 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.444574 24490 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:54.485925 24480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.014s sys 0.030s
W20251215 08:59:54.446758 24486 log.cc:927] Time spent T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20251215 08:59:54.354606 24475 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.464799 24161 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.509151 24439 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:54.468335 24236 leader_election.cc:290] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:54.473773 24215 raft_consensus.cc:3060] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:54.474318 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:49730 (request call id 1634) took 1689 ms. Trace:
I20251215 08:59:54.476500 24085 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.343372 24141 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.552189 24161 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.440605 24441 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251215 08:59:54.557401 24085 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 11 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.489307 21339 pending_rounds.cc:107] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251215 08:59:54.489393 21379 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:54.571834 21339 raft_consensus.cc:2889] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251215 08:59:54.573616 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.358850 24476 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.498521 21259 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
W20251215 08:59:54.483800 24483 log.cc:927] Time spent T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.404s user 0.001s sys 0.000s
W20251215 08:59:54.509405 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 224812us
W20251215 08:59:54.581705 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 92386.6us
I20251215 08:59:54.408697 24304 raft_consensus.cc:493] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.519800 21261 raft_consensus.cc:1275] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251215 08:59:54.583781 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.585410 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:54.586644 21261 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20251215 08:59:54.554160 24426 consensus_meta.cc:287] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.522s user 0.010s sys 0.000s
W20251215 08:59:54.445763 24488 log.cc:927] Time spent T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20251215 08:59:54.409037 24341 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:54.520824 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.356863 24427 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:54.527541 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:54.591487 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.530284 24505 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:54.531376 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 249845us
I20251215 08:59:54.531651 24500 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:54.382643 24414 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.130s
I20251215 08:59:54.614960 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.615459 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:54.459403 24449 log.cc:927] Time spent T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 1.157s user 0.011s sys 0.000s
I20251215 08:59:54.508595 24343 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
W20251215 08:59:54.548314 21230 leader_election.cc:336] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.091s (SENT)
W20251215 08:59:54.522953 24480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s user 0.014s sys 0.031s
I20251215 08:59:54.508821 24423 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:54.512151 24268 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.640365 24423 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.552868 21341 rpcz_store.cc:276] 1215 08:59:52.785215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.082517 (+297302us) service_pool.cc:225] Handling call
1215 08:59:54.473517 (+1391000us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400163d54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1e0c0 00007f3987bbcbba 00007f398cb6d58e 00007f398610c437 00007f398610c3c2 00007f398610c2f1 00007f398610c26c 00007f398610b522 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:54.474179 (+ 662us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245211008}
I20251215 08:59:54.534457 24371 raft_consensus.cc:2749] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:54.668201 24423 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.669847 21265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 377) took 1319 ms. Trace:
I20251215 08:59:54.670272 21265 rpcz_store.cc:276] 1215 08:59:53.350107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.375009 (+ 24902us) service_pool.cc:225] Handling call
1215 08:59:53.375458 (+ 449us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:53.969791 (+594333us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.969836 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:54.001728 (+ 31892us) log.cc:844] Serialized 26 byte log entry
1215 08:59:54.386259 (+384531us) raft_consensus.cc:1645] Marking committed up to 6
1215 08:59:54.386313 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.386404 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:54.669638 (+283234us) raft_consensus.cc:1685] finished
1215 08:59:54.669656 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:54.669786 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16180,"raft.queue_time_us":729436,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":877657,"threads_started":2,"wal-append.queue_time_us":521450}
I20251215 08:59:54.671921 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:54.672403 21944 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.007s sys 0.043s
W20251215 08:59:54.672679 21944 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.779s user 0.007s sys 0.043s
I20251215 08:59:54.556738 24161 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:54.580888 21260 raft_consensus.cc:2393] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 8: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:54.548692 21232 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:54.675547 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 126920us
I20251215 08:59:54.584113 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.678079 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.679584 21980 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:54.680995 24472 raft_consensus.cc:2804] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 08:59:54.681576 24472 raft_consensus.cc:697] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:54.682408 24472 consensus_queue.cc:237] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.594270 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:54.672928 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 779239us
I20251215 08:59:54.684010 21971 raft_consensus.cc:2393] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:54.612732 21972 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.580535 21259 pending_rounds.cc:85] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 4
I20251215 08:59:54.523094 24409 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.613472 21978 raft_consensus.cc:1240] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:54.595494 21977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 684) took 5070 ms. Trace:
W20251215 08:59:54.587961 24496 log.cc:927] Time spent T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.066s user 0.003s sys 0.000s
I20251215 08:59:54.586831 24104 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:54.535441 24490 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:54.621532 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 73271.1us
I20251215 08:59:54.692073 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:54.695859 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.628782 21229 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:54.632112 24150 consensus_queue.cc:1048] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251215 08:59:54.642697 24507 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20251215 08:59:54.612017 24341 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:54.647881 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:54.589908 24304 raft_consensus.cc:515] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.632776 24085 consensus_queue.cc:1059] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:54.654560 24395 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:54.655635 24427 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:54.656275 24465 leader_election.cc:290] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:54.730037 24493 log.cc:927] Time spent T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.068s user 0.003s sys 0.000s
I20251215 08:59:54.658275 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.731659 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.731698 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.732668 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.658592 21406 raft_consensus.cc:1240] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
I20251215 08:59:54.733948 21264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2151) took 4711 ms. Trace:
I20251215 08:59:54.734369 21264 rpcz_store.cc:276] 1215 08:59:50.022244 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:50.022419 (+ 175us) service_pool.cc:225] Handling call
1215 08:59:50.066586 (+ 44167us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:54.337497 (+4270911us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b18003fc220. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f398775b85c 00007f3984ae8ffb 00007f398d8d07e9 00007f3984b0528b 00007f3984b0352f 00007f3984b03045 00007f3987babce0 00007f3987bac87b 00007f3987bb72cb
1215 08:59:54.337622 (+ 125us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:54.337647 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:54.338138 (+ 491us) log.cc:844] Serialized 26 byte log entry
1215 08:59:54.533518 (+195380us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:54.533617 (+ 99us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.533979 (+ 362us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:54.733671 (+199692us) raft_consensus.cc:1685] finished
1215 08:59:54.733692 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:54.733877 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32218,"raft.queue_time_us":131594,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":15275,"spinlock_wait_cycles":13404928,"thread_start_us":349396,"threads_started":2,"wal-append.queue_time_us":194544}
I20251215 08:59:54.736373 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.660380 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.619690 24505 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.632833 24501 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:54.665033 24475 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.739406 24505 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:54.663337 24268 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
W20251215 08:59:54.666155 24369 consensus_meta.cc:287] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.592s user 0.008s sys 0.002s
I20251215 08:59:54.685367 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.685716 21259 pending_rounds.cc:107] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20251215 08:59:54.741081 21259 raft_consensus.cc:2889] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:54.741369 21407 raft_consensus.cc:1240] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:54.589946 21379 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:54.741942 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:54.699936 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 71219.2us
I20251215 08:59:54.714514 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:54.743108 21260 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:54.743706 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.744446 21259 raft_consensus.cc:2368] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7.
I20251215 08:59:54.744961 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.745688 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.746088 21259 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 3.
I20251215 08:59:54.746623 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:54.747170 21259 raft_consensus.cc:2393] T 641cd3a624114cf98d427a39378b6da6 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 6: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:54.715788 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:54.718694 21974 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:54.597731 24426 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.734038 21266 raft_consensus.cc:1240] T f757e923d11a495091c94621efa6e4c1 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:54.740623 21415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59102 (request call id 326) took 1393 ms (client timeout 1541 ms). Trace:
W20251215 08:59:54.755470 21415 rpcz_store.cc:269] 1215 08:59:53.346859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.392050 (+ 45191us) service_pool.cc:225] Handling call
1215 08:59:54.740436 (+1348386us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b54000bcf54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:54.740570 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280699776}
I20251215 08:59:54.673410 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 1950) took 3552 ms. Trace:
I20251215 08:59:54.732832 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:54.688175 21977 rpcz_store.cc:276] 1215 08:59:49.518773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:49.520205 (+ 1432us) service_pool.cc:225] Handling call
1215 08:59:49.520480 (+ 275us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:54.065169 (+4544689us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.588697 (+523528us) spinlock_profiling.cc:243] Waited 4.54 s on lock 0x7b54003f11d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.588939 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10447994624}
I20251215 08:59:54.751684 21978 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
I20251215 08:59:54.757037 21412 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
W20251215 08:59:54.756740 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 166865us
I20251215 08:59:54.757468 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.756743 21334 rpcz_store.cc:276] 1215 08:59:51.120613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.032136 (+1911523us) service_pool.cc:225] Handling call
1215 08:59:53.032475 (+ 339us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:53.032698 (+ 223us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.032721 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:53.033182 (+ 461us) log.cc:844] Serialized 26 byte log entry
1215 08:59:53.090200 (+ 57018us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:53.090255 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:53.090352 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:54.458973 (+1368621us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b240001f4a8. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f398495a9a4 00007f3984958aab 00007f39849e79a9 00007f39849e7635 00007f39849e6c7d 00007f39877374ae 00007f398774ec2c 00007f398774df56 00007f398774db78 00007f39877575c1
1215 08:59:54.673073 (+214100us) raft_consensus.cc:1685] finished
1215 08:59:54.673118 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:54.673351 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14993280,"thread_start_us":56339,"threads_started":1,"wal-append.queue_time_us":158714}
I20251215 08:59:54.765288 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:54.718168 24303 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:54.668224 24481 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:54.622138 24500 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:54.756049 24369 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:54.770251 24258 consensus_meta.cc:287] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.668s user 0.011s sys 0.000s
I20251215 08:59:54.814481 23959 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20251215 08:59:54.766203 24310 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:54.923216 21414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 332) took 1535 ms. Trace:
I20251215 08:59:54.960875 21414 rpcz_store.cc:276] 1215 08:59:53.388040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.557888 (+169848us) service_pool.cc:225] Handling call
1215 08:59:53.896203 (+338315us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:54.088805 (+192602us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.791714 (+702909us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b540026cad4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.923147 (+131433us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":431472000}
I20251215 08:59:54.754878 24427 consensus_queue.cc:1048] T f14cfff8c5a44ad78299ada0ca53baa9 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.039s
I20251215 08:59:54.767201 24475 consensus_queue.cc:1059] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20251215 08:59:54.739979 24505 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:54.792858 24437 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.767095 24247 raft_consensus.cc:2749] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:54.924803 21410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 325) took 1579 ms. Trace:
W20251215 08:59:54.924878 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 164769us
I20251215 08:59:54.782356 24465 consensus_queue.cc:1048] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20251215 08:59:54.927448 24371 consensus_queue.cc:1059] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:54.929473 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:55.022125 21332 raft_consensus.cc:2376] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7. Re-sending same reply.
I20251215 08:59:54.937588 21263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2183) took 2031 ms. Trace:
I20251215 08:59:55.023087 21263 rpcz_store.cc:276] 1215 08:59:52.905915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.906098 (+ 183us) service_pool.cc:225] Handling call
1215 08:59:52.906487 (+ 389us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.058989 (+152502us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:53.059016 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:53.059055 (+ 39us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:53.059085 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.937301 (+1878216us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54000cce54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.937364 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:54.937521 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":350259968}
W20251215 08:59:54.932883 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 172547us
I20251215 08:59:55.024359 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:55.024927 21263 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:55.027740 24304 leader_election.cc:290] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:55.028395 21379 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.029219 24304 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:55.029785 24304 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.032714 24304 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.041571 21979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 697) took 1697 ms. Trace:
I20251215 08:59:54.949784 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.956077 21337 raft_consensus.cc:1275] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20251215 08:59:55.042335 21337 pending_rounds.cc:85] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 3
I20251215 08:59:55.042614 21337 pending_rounds.cc:107] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20251215 08:59:55.042830 21337 raft_consensus.cc:2889] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:54.978484 21976 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:55.044533 24440 raft_consensus.cc:3060] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:55.044799 21260 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 8.
I20251215 08:59:54.977720 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:54.959416 23974 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:54.998808 24303 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:54.999447 24104 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:55.001662 24141 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.006328 24426 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:55.008663 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.009368 24512 raft_consensus.cc:493] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:55.009822 24215 raft_consensus.cc:493] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:54.971747 24476 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:54.953052 24439 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.017868 24256 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:55.018529 24369 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.018941 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:55.019697 24508 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:54.982784 24258 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:55.021209 21410 rpcz_store.cc:276] 1215 08:59:53.345816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.346027 (+ 211us) service_pool.cc:225] Handling call
1215 08:59:53.363869 (+ 17842us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:53.959965 (+596096us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:54.798684 (+838719us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b54001761d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:54.924712 (+126028us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1256960256}
I20251215 08:59:55.035530 24343 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.036950 24409 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.037503 24437 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.061966 24256 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.070271 24465 raft_consensus.cc:3055] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 08:59:55.070703 24465 raft_consensus.cc:740] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:55.071547 24465 consensus_queue.cc:260] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.053642 24476 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.071805 24256 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.038764 24505 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.039767 24475 consensus_queue.cc:1059] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:54.973635 24274 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
I20251215 08:59:55.041935 21979 rpcz_store.cc:276] 1215 08:59:53.344350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.344510 (+ 160us) service_pool.cc:225] Handling call
1215 08:59:53.344894 (+ 384us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:54.224891 (+879997us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:55.041345 (+816454us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b54003fe3d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:55.041508 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2012972800}
I20251215 08:59:54.756940 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:55.082448 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:55.083252 21259 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Advancing to term 10
W20251215 08:59:55.050565 21304 connection.cc:232] RPC call timeout handler was delayed by -1.7661797550000529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15380000000000002s for a response
I20251215 08:59:55.053422 24512 raft_consensus.cc:515] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.059032 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.066974 21261 raft_consensus.cc:2468] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 9.
I20251215 08:59:55.076682 21973 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:55.050020 24490 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.090945 21264 raft_consensus.cc:2368] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:55.093317 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.089814 21304 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:55.096246 21979 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:55.096786 21410 raft_consensus.cc:1275] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20251215 08:59:55.098841 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.075445 24505 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.129829 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.132382 21411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:55.132823 21411 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:55.087457 24475 raft_consensus.cc:3055] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:55.133963 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.134490 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:55.134543 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.135211 21414 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Advancing to term 11
I20251215 08:59:55.135262 21410 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 10.
I20251215 08:59:55.135965 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.136260 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.136551 21406 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:55.072978 24437 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.136965 21410 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9.
I20251215 08:59:55.135262 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:55.131742 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.137847 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.074774 24476 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.089901 24268 leader_election.cc:290] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:55.077054 24508 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.075965 24258 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.078208 24215 raft_consensus.cc:515] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.080966 24481 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:55.139094 24475 raft_consensus.cc:740] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:55.158339 24500 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.159718 24512 leader_election.cc:290] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.160804 24300 raft_consensus.cc:3055] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Stepping down as leader of term 8
I20251215 08:59:55.161280 24300 raft_consensus.cc:740] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:55.162257 24300 consensus_queue.cc:260] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.163774 24300 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:55.166496 24258 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.168646 24508 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.169607 24475 consensus_queue.cc:260] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.170105 24343 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.094491 24141 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.061s
I20251215 08:59:55.170981 24475 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:55.171932 24341 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.173530 24489 raft_consensus.cc:3055] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:55.173964 24489 raft_consensus.cc:740] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:55.174798 24489 consensus_queue.cc:260] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.183177 24439 leader_election.cc:290] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:55.187310 24360 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.004s sys 0.024s
W20251215 08:59:55.187804 24360 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s user 0.004s sys 0.024s
W20251215 08:59:55.235644 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.235761 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.241704 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.244097 21263 raft_consensus.cc:1240] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20251215 08:59:55.246081 21263 raft_consensus.cc:1240] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 4. Current term is 5. Ops: []
W20251215 08:59:55.250201 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 160338us
W20251215 08:59:55.250404 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 161878us
W20251215 08:59:55.253973 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.255008 21307 connection.cc:232] RPC call timeout handler was delayed by -2.1070201239999733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
W20251215 08:59:55.255654 21307 connection.cc:232] RPC call timeout handler was delayed by -1.969320123999978s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20251215 08:59:55.256042 21307 connection.cc:232] RPC call timeout handler was delayed by -1.7479201239999611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
W20251215 08:59:55.256459 21307 connection.cc:232] RPC call timeout handler was delayed by -1.5904201240000475s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.215s for a response
I20251215 08:59:55.256367 21378 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:55.240564 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:55.259867 21380 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.260135 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.261174 21406 raft_consensus.cc:2468] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:55.262838 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:55.256943 21307 connection.cc:232] RPC call timeout handler was delayed by -1.028820124000049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27740000000000004s for a response
I20251215 08:59:55.264642 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:55.265751 21230 leader_election.cc:336] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.935s (SENT)
I20251215 08:59:55.266088 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.266486 21230 leader_election.cc:304] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:55.267556 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.269178 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.270424 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.139321 21410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.270471 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.271643 21307 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:55.270768 21263 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20251215 08:59:55.268731 21230 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.940s (SENT)
I20251215 08:59:55.258548 21261 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7.
I20251215 08:59:55.139355 21412 raft_consensus.cc:2393] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:55.138887 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.333400 24104 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:55.283634 21259 raft_consensus.cc:2468] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 10.
I20251215 08:59:55.292433 21414 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 11.
I20251215 08:59:55.334234 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.335353 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.335547 21259 raft_consensus.cc:1240] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:55.336054 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.337083 21414 raft_consensus.cc:1240] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:55.339080 21410 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251215 08:59:55.296279 21411 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
W20251215 08:59:55.298768 21944 leader_election.cc:336] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.562s (SENT)
W20251215 08:59:55.341313 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 72684.2us
I20251215 08:59:55.342907 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:55.343217 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 44772.3us
I20251215 08:59:55.334230 24104 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:55.343966 21230 leader_election.cc:336] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.942s (SENT)
I20251215 08:59:55.344446 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:55.316361 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 44908.3us
I20251215 08:59:55.316686 21406 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:55.317917 24512 raft_consensus.cc:493] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:55.192180 24482 raft_consensus.cc:493] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:55.334280 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.346390 24512 raft_consensus.cc:515] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.347018 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:55.346845 24482 raft_consensus.cc:515] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:55.318753 21971 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.004s sys 0.015s
I20251215 08:59:55.318936 24520 raft_consensus.cc:493] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:55.319860 24523 raft_consensus.cc:493] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:55.320712 24215 leader_election.cc:290] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.321264 21409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:55.319660 21262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 375) took 2003 ms. Trace:
I20251215 08:59:55.330363 24069 consensus_queue.cc:1048] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251215 08:59:55.331035 24219 raft_consensus.cc:2749] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251215 08:59:55.347218 21944 leader_election.cc:336] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.722s (SENT)
I20251215 08:59:55.317407 24508 consensus_queue.cc:1059] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251215 08:59:55.320905 21266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1553) took 4934 ms. Trace:
I20251215 08:59:55.355245 24520 raft_consensus.cc:515] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:55.360662 21971 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s user 0.004s sys 0.015s
I20251215 08:59:55.371147 21409 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 9: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:55.371766 24523 raft_consensus.cc:515] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.372514 21307 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:55.373857 24500 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:55.356904 21266 rpcz_store.cc:276] 1215 08:59:50.386198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:50.388307 (+ 2109us) service_pool.cc:225] Handling call
1215 08:59:54.733864 (+4345557us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:54.770382 (+ 36518us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:55.320473 (+550091us) spinlock_profiling.cc:243] Waited 4.34 s on lock 0x7b5400323f50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:55.320721 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9993177856}
I20251215 08:59:55.379302 24258 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:55.385715 24360 consensus_queue.cc:1048] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.002s
I20251215 08:59:55.387161 24520 leader_election.cc:290] T bfcf2c8cd22a488aa741f1cede6e86ef P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.387306 24508 raft_consensus.cc:2749] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:55.391742 24440 raft_consensus.cc:2749] T bfcf2c8cd22a488aa741f1cede6e86ef P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:55.393707 21257 raft_consensus.cc:2410] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:55.352463 21262 rpcz_store.cc:276] 1215 08:59:53.316325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:53.324983 (+ 8658us) service_pool.cc:225] Handling call
1215 08:59:53.325426 (+ 443us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:54.060196 (+734770us) raft_consensus.cc:1534] Early marking committed up to index 9
1215 08:59:54.065579 (+ 5383us) log.cc:844] Serialized 14 byte log entry
1215 08:59:54.235188 (+169609us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:54.236056 (+ 868us) raft_consensus.cc:1645] Marking committed up to 9
1215 08:59:54.236101 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:55.318492 (+1082391us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b1800211540. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984ad3fe6 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49
1215 08:59:55.319150 (+ 658us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b54000cd854. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:55.319193 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:55.319418 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":5601,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":766,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":35321,"spinlock_wait_cycles":1777706752,"thread_start_us":169276,"threads_started":1,"wal-append.queue_time_us":421162}
I20251215 08:59:55.403046 21408 raft_consensus.cc:1240] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:55.404971 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 79751.4us
W20251215 08:59:55.405104 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 58784.4us
I20251215 08:59:55.405978 24064 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:55.408550 24219 consensus_queue.cc:1048] T ac8397b87ffc453b8f468eee77a8e410 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20251215 08:59:55.414372 21974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46154 (request call id 726) took 2685 ms (client timeout 1845 ms). Trace:
W20251215 08:59:55.414837 21974 rpcz_store.cc:269] 1215 08:59:52.728800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:52.729142 (+ 342us) service_pool.cc:225] Handling call
1215 08:59:55.413908 (+2684766us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b5400570154. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:55.414127 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4512346368}
I20251215 08:59:55.419370 24302 raft_consensus.cc:3055] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:55.419832 24302 raft_consensus.cc:740] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:55.421622 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.422382 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.423244 21976 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:55.423889 21337 raft_consensus.cc:1275] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:55.425307 21261 raft_consensus.cc:1240] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 9. Current term is 10. Ops: []
I20251215 08:59:55.424880 24302 consensus_queue.cc:260] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.426746 24302 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:55.427237 24440 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:55.428264 24440 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } attrs { replace: true } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.387182 21307 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:55.430688 24104 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:55.431404 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 44495.5us
I20251215 08:59:55.432247 24440 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.434144 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.435061 21974 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 2.
I20251215 08:59:55.230647 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:55.387940 24236 consensus_queue.cc:1048] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20251215 08:59:55.432358 21307 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:55.430015 21258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2200) took 1250 ms. Trace:
I20251215 08:59:55.408550 21381 leader_election.cc:304] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
W20251215 08:59:55.444082 21944 leader_election.cc:336] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.830s (SENT)
I20251215 08:59:55.389596 24258 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 LEADER]: Becoming Leader. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:55.452557 24508 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:55.455506 24469 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:55.456117 24469 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:55.456529 24469 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:55.465684 21944 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:55.466809 24236 raft_consensus.cc:2749] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20251215 08:59:55.468499 21944 leader_election.cc:336] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.856s (SENT)
I20251215 08:59:55.473928 24258 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.440846 21258 rpcz_store.cc:276] 1215 08:59:54.179257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:54.240676 (+ 61419us) service_pool.cc:225] Handling call
1215 08:59:54.244178 (+ 3502us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:54.660904 (+416726us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:54.660931 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:54.661406 (+ 475us) log.cc:844] Serialized 26 byte log entry
1215 08:59:54.926030 (+264624us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:54.926069 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:55.429458 (+503389us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b540020b554. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:55.429530 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:55.429582 (+ 52us) raft_consensus.cc:1685] finished
1215 08:59:55.429595 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:55.429834 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1011,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":895491328,"thread_start_us":264274,"threads_started":1,"wal-append.queue_time_us":360610}
W20251215 08:59:55.483557 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 139724us
I20251215 08:59:55.484369 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:55.485941 24371 raft_consensus.cc:2804] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251215 08:59:55.486375 24371 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:55.486428 24523 leader_election.cc:290] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.486898 24371 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Advancing to term 11
I20251215 08:59:55.492846 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.493144 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.475943 24512 leader_election.cc:290] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:55.502871 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 94643.4us
I20251215 08:59:55.512564 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:55.513533 21341 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:55.514465 21257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50710 (request call id 2207) took 1270 ms. Trace:
I20251215 08:59:55.515153 21262 raft_consensus.cc:1275] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20251215 08:59:55.515671 21262 pending_rounds.cc:85] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Aborting all ops after (but not including) 7
I20251215 08:59:55.516121 21262 pending_rounds.cc:107] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20251215 08:59:55.516435 21262 raft_consensus.cc:2889] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:55.515501 24361 raft_consensus.cc:2804] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader election won for term 10
I20251215 08:59:55.520149 21976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:55.524861 21976 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:55.526010 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.514916 21257 rpcz_store.cc:276] 1215 08:59:54.243902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:54.307574 (+ 63672us) service_pool.cc:225] Handling call
1215 08:59:55.514092 (+1206518us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001342d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:55.514390 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2391032064}
I20251215 08:59:55.528283 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.535548 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2210) took 1210 ms. Trace:
I20251215 08:59:55.536053 21260 rpcz_store.cc:276] 1215 08:59:54.325043 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:55.054187 (+729144us) service_pool.cc:225] Handling call
1215 08:59:55.054639 (+ 452us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:55.063406 (+ 8767us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:55.063450 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:55.077481 (+ 14031us) log.cc:844] Serialized 26 byte log entry
1215 08:59:55.445170 (+367689us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:55.447575 (+ 2405us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:55.451536 (+ 3961us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:55.535131 (+ 83595us) raft_consensus.cc:1685] finished
1215 08:59:55.535153 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:55.535342 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7262,"raft.queue_time_us":8427,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":12133,"thread_start_us":125161,"threads_started":1,"wal-append.queue_time_us":150612}
I20251215 08:59:55.537369 21229 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:55.540176 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:55.541545 24248 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:55.541989 21259 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
W20251215 08:59:55.543255 24310 consensus_meta.cc:287] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e: Time spent flushing consensus metadata: real 0.571s user 0.014s sys 0.000s
I20251215 08:59:55.543781 24310 raft_consensus.cc:2749] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:55.544848 21260 raft_consensus.cc:1275] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251215 08:59:55.545660 24372 raft_consensus.cc:2804] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:55.546197 24372 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:55.546641 24372 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:55.551354 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 119203us
I20251215 08:59:55.552762 21307 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:55.554301 24514 consensus_queue.cc:1059] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:55.555030 21307 leader_election.cc:400] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7.
I20251215 08:59:55.555426 21307 leader_election.cc:403] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251215 08:59:55.555902 24514 raft_consensus.cc:2749] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 08:59:55.557251 24523 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 6. Current term is 7.
I20251215 08:59:55.575784 21259 raft_consensus.cc:1275] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:55.582882 21265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2208) took 1275 ms. Trace:
I20251215 08:59:55.583266 21265 rpcz_store.cc:276] 1215 08:59:54.296868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:54.677238 (+380370us) service_pool.cc:225] Handling call
1215 08:59:55.046450 (+369212us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b5800002498. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984ac8ac1 00007f3984ac8972 00007f398d971069 00007f398caf3260 00007f398cbbbdbb 00007f398cbbbe41 00007f398cb7daf0 00007f398cb68459 00007f398cb67f02 00007f39860f4d34
1215 08:59:55.046634 (+ 184us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:55.049181 (+ 2547us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:55.049223 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:55.049740 (+ 517us) log.cc:844] Serialized 26 byte log entry
1215 08:59:55.447746 (+398006us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:55.471631 (+ 23885us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:55.482424 (+ 10793us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:55.520623 (+ 38199us) raft_consensus.cc:1685] finished
1215 08:59:55.520636 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:55.572362 (+ 51726us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2162,"raft.queue_time_us":3913,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":5893,"spinlock_wait_cycles":3636992,"thread_start_us":326741,"threads_started":1,"wal-append.queue_time_us":456719}
I20251215 08:59:55.571198 24361 raft_consensus.cc:697] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:55.591180 24361 consensus_queue.cc:237] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.579164 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:55.584851 21944 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.058s (SENT)
I20251215 08:59:55.597199 21974 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 9.
I20251215 08:59:55.603367 24465 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Advancing to term 6
W20251215 08:59:55.608124 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 76989.7us
W20251215 08:59:55.618911 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 76815.7us
I20251215 08:59:55.556864 24514 raft_consensus.cc:3055] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Stepping down as leader of term 4
I20251215 08:59:55.627386 21333 raft_consensus.cc:2410] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20251215 08:59:55.617599 21307 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:55.622862 24514 raft_consensus.cc:740] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:55.627377 24215 raft_consensus.cc:3055] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Stepping down as leader of term 6
W20251215 08:59:55.652953 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.658427 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 90431.1us
I20251215 08:59:55.676127 21232 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:55.679531 24150 consensus_queue.cc:1059] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20251215 08:59:55.681294 24500 raft_consensus.cc:2804] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251215 08:59:55.681754 24500 raft_consensus.cc:493] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:55.682082 24500 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:55.693981 24489 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:55.709228 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:55.711294 21946 leader_election.cc:400] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:55.711647 21946 leader_election.cc:403] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251215 08:59:55.712105 24248 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:55.712533 24248 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:55.712843 24248 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:55.713806 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 43581.9us
I20251215 08:59:55.661239 24215 raft_consensus.cc:740] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
W20251215 08:59:55.718829 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.720489 24274 consensus_meta.cc:287] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.625s user 0.013s sys 0.000s
I20251215 08:59:55.721148 24339 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:55.721731 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 74868us
I20251215 08:59:55.721824 24339 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.723345 21232 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:55.724182 21379 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:55.724495 24439 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251215 08:59:55.725142 24439 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:55.725636 24439 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Advancing to term 10
W20251215 08:59:55.726006 24475 consensus_meta.cc:287] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.554s user 0.011s sys 0.005s
I20251215 08:59:55.726773 24475 raft_consensus.cc:2804] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:55.727254 24475 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:55.727676 24475 raft_consensus.cc:3060] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:55.731027 24371 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.732721 24064 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20251215 08:59:55.734448 24357 consensus_queue.cc:1048] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251215 08:59:55.736220 24531 log.cc:927] Time spent T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20251215 08:59:55.760871 24339 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.764176 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.767273 24300 consensus_queue.cc:1048] T 0c3a05444c334cad8e7562a15e4d68ba P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20251215 08:59:55.739367 24469 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.773249 24215 consensus_queue.cc:260] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.774416 24469 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.739802 24421 raft_consensus.cc:493] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:55.775260 21337 raft_consensus.cc:1240] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251215 08:59:55.775285 24421 raft_consensus.cc:515] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:55.776875 21338 raft_consensus.cc:1275] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251215 08:59:55.777515 24421 leader_election.cc:290] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:55.777983 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b83855234bda47a1b06c3f932728766b" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:55.746790 21340 raft_consensus.cc:1275] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251215 08:59:55.779418 21338 raft_consensus.cc:3055] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:55.779692 21338 raft_consensus.cc:740] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:55.780893 21333 raft_consensus.cc:1275] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20251215 08:59:55.781515 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.781558 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 107526us
I20251215 08:59:55.785883 24357 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:55.746955 24439 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.751128 24514 consensus_queue.cc:260] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:55.752310 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 167509us
W20251215 08:59:55.675097 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.767112 24372 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.737457 21381 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:55.772208 24064 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:55.774706 24215 raft_consensus.cc:3060] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:55.803014 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 78275.3us
I20251215 08:59:55.803531 21260 raft_consensus.cc:1240] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 11 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 10. Current term is 11. Ops: []
I20251215 08:59:55.812678 21340 raft_consensus.cc:1275] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20251215 08:59:55.814471 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.815148 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:55.812285 21230 leader_election.cc:336] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.549s (SENT)
I20251215 08:59:55.822561 21230 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:55.828855 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.829185 21974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.787416 24482 leader_election.cc:290] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:55.830824 21304 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.829057 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.807135 24310 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:55.843498 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 119522us
I20251215 08:59:55.844610 21307 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:55.850137 21976 raft_consensus.cc:2468] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 3.
W20251215 08:59:55.857863 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.862164 21164 catalog_manager.cc:5654] T d20229ce83d84c8ba85a5fc4604d817d P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 3 to 4, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 4 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:55.823328 24357 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.865114 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:55.865696 24177 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:55.865892 21979 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:55.866297 24177 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:55.867000 24357 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.842756 21338 consensus_queue.cc:260] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:55.865275 24333 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251215 08:59:55.870869 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 133501us
I20251215 08:59:55.880733 21229 leader_election.cc:304] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:55.884039 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 71817.9us
I20251215 08:59:55.895938 21379 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:55.868323 21164 catalog_manager.cc:5654] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 4 to 5, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 5 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
W20251215 08:59:55.888382 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 66649.7us
I20251215 08:59:55.959950 21338 raft_consensus.cc:3060] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:55.962530 21944 leader_election.cc:336] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.168s (SENT)
W20251215 08:59:55.950229 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 63172us
I20251215 08:59:55.962968 21944 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:55.903447 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 50565.3us
W20251215 08:59:55.961512 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 80867.4us
I20251215 08:59:55.872607 24439 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.917876 24512 consensus_queue.cc:1059] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20251215 08:59:55.971596 24085 raft_consensus.cc:2749] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:55.971647 24421 raft_consensus.cc:2749] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:55.972702 21304 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
W20251215 08:59:55.973122 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 84929.2us
I20251215 08:59:55.969323 21229 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:55.981568 24409 raft_consensus.cc:2749] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193) with higher term. Message: Invalid argument: T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:55.868949 24177 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.879877 24371 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 11 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:55.976003 21164 catalog_manager.cc:5654] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 3 to 6, leader changed from 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67) to f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68). New cstate: current_term: 6 leader_uuid: "f0824fa5450345719cac22fc6ae3ce2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } } }
I20251215 08:59:55.950685 24469 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:55.985813 24514 raft_consensus.cc:3060] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Advancing to term 5
I20251215 08:59:55.986397 24357 raft_consensus.cc:2804] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251215 08:59:55.994458 24535 log.cc:927] Time spent T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.302s user 0.001s sys 0.004s
W20251215 08:59:56.018458 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:55.871909 24349 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20251215 08:59:55.880017 24310 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:56.030822 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 83653.4us
I20251215 08:59:56.031920 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:56.028680 21304 leader_election.cc:304] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:55.996482 24469 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:56.013183 24274 raft_consensus.cc:2804] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 08:59:56.016222 24482 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:56.047482 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:56.053236 24520 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:56.053680 24520 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:56.054049 24520 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:56.055222 21164 catalog_manager.cc:5654] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 2 to 6, leader changed from f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 6 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251215 08:59:56.058372 21164 catalog_manager.cc:5654] T 0922c97d6dce49f0a3b98d200f4f651e P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 4 to 5, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 5 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:56.059688 24248 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.062827 21307 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:56.063645 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 149055us
W20251215 08:59:56.064369 21944 leader_election.cc:336] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.186s (SENT)
I20251215 08:59:56.020589 24357 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:56.065310 24357 raft_consensus.cc:3060] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:56.066847 21978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:56.067059 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:56.067679 21971 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:56.069904 24310 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:56.045953 24482 raft_consensus.cc:493] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:56.075423 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 59921.7us
I20251215 08:59:56.077976 21260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 389) took 1020 ms. Trace:
I20251215 08:59:56.078522 21979 raft_consensus.cc:1240] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:56.078366 21260 rpcz_store.cc:276] 1215 08:59:55.057805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:55.546048 (+488243us) service_pool.cc:225] Handling call
1215 08:59:55.546405 (+ 357us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:55.821352 (+274947us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:56.077739 (+256387us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54002e54d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:56.077916 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":590801664}
I20251215 08:59:56.031183 24258 raft_consensus.cc:3055] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:56.040675 24372 leader_election.cc:290] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:56.045267 24469 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:56.094007 21338 raft_consensus.cc:1275] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20251215 08:59:56.094558 21338 pending_rounds.cc:85] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 4
I20251215 08:59:56.116220 21338 pending_rounds.cc:107] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20251215 08:59:56.116703 21338 raft_consensus.cc:2889] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251215 08:59:56.147738 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 76819.9us
W20251215 08:59:56.147647 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 45171us
I20251215 08:59:56.084930 24482 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:56.150128 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 73687.4us
I20251215 08:59:56.029965 24475 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.085734 24258 raft_consensus.cc:740] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:56.089355 24274 raft_consensus.cc:697] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [term 3 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:56.164115 24523 raft_consensus.cc:2804] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251215 08:59:56.197578 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 164056us
I20251215 08:59:56.198590 24393 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:56.199287 24393 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:56.200062 24542 log.cc:927] Time spent T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
W20251215 08:59:56.199683 21161 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36408 (request call id 213) took 28403 ms (client timeout 15000 ms). Trace:
I20251215 08:59:56.210862 21379 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:56.193251 24523 raft_consensus.cc:493] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:56.185177 21164 catalog_manager.cc:5654] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 reported cstate change: term changed from 5 to 6, leader changed from 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66) to beefe15185ac42ef9a20251007f39a56 (127.19.163.65). New cstate: current_term: 6 leader_uuid: "beefe15185ac42ef9a20251007f39a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } health_report { overall_health: UNKNOWN } } }
I20251215 08:59:56.182909 24512 raft_consensus.cc:2804] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251215 08:59:56.221411 24274 consensus_queue.cc:237] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:56.224468 24532 log.cc:927] Time spent T 01ad5646d4d04a6a996ce407925b8650 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.276s user 0.002s sys 0.000s
W20251215 08:59:56.230357 24541 log.cc:927] Time spent T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.244s user 0.001s sys 0.001s
W20251215 08:59:56.202291 21161 rpcz_store.cc:269] 1215 08:59:27.669465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:27.669779 (+ 314us) service_pool.cc:225] Handling call
1215 08:59:56.044713 (+28374934us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1215 08:59:48.698439 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1215 08:59:48.699009 (+ 570us) write_op.cc:432] Acquiring schema lock in shared mode
1215 08:59:48.699051 (+ 42us) write_op.cc:435] Acquired schema lock
1215 08:59:48.699068 (+ 17us) tablet.cc:660] Decoding operations
1215 08:59:48.699836 (+ 768us) write_op.cc:620] Acquiring the partition lock for write op
1215 08:59:48.699914 (+ 78us) write_op.cc:641] Partition lock acquired for write op
1215 08:59:48.699946 (+ 32us) tablet.cc:683] Acquiring locks for 5 operations
1215 08:59:48.706716 (+ 6770us) tablet.cc:699] Row locks acquired
1215 08:59:48.706740 (+ 24us) write_op.cc:260] PREPARE: finished
1215 08:59:48.706908 (+ 168us) write_op.cc:270] Start()
1215 08:59:48.707052 (+ 144us) write_op.cc:276] Timestamp: P: 1765789188706858 usec, L: 0
1215 08:59:48.707072 (+ 20us) op_driver.cc:348] REPLICATION: starting
1215 08:59:48.707816 (+ 744us) log.cc:844] Serialized 2147 byte log entry
1215 08:59:49.589146 (+881330us) op_driver.cc:464] REPLICATION: finished
1215 08:59:49.947228 (+358082us) write_op.cc:301] APPLY: starting
1215 08:59:49.956938 (+ 9710us) tablet.cc:1365] starting BulkCheckPresence
1215 08:59:49.997260 (+ 40322us) tablet.cc:1368] finished BulkCheckPresence
1215 08:59:50.017438 (+ 20178us) tablet.cc:1370] starting ApplyRowOperation cycle
1215 08:59:50.231000 (+213562us) tablet.cc:1381] finished ApplyRowOperation cycle
1215 08:59:50.231205 (+ 205us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
1215 08:59:50.231277 (+ 72us) write_op.cc:312] APPLY: finished
1215 08:59:50.231560 (+ 283us) log.cc:844] Serialized 47 byte log entry
1215 08:59:50.242072 (+ 10512us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:50.242253 (+ 181us) write_op.cc:454] Released schema lock
1215 08:59:50.242314 (+ 61us) write_op.cc:341] FINISH: Updating metrics
1215 08:59:50.242877 (+ 563us) write_op.cc:489] Releasing partition, row and schema locks
1215 08:59:50.242917 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":894552,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":316221,"apply.run_cpu_time_us":4044,"apply.run_wall_time_us":309038,"num_ops":5,"prepare.queue_time_us":1060997,"prepare.run_cpu_time_us":127966,"prepare.run_wall_time_us":649335,"raft.queue_time_us":106230,"raft.run_cpu_time_us":84453,"raft.run_wall_time_us":276667,"replication_time_us":873495,"thread_start_us":1250514,"threads_started":4,"wal-append.queue_time_us":666521,"wal-append.run_cpu_time_us":19967,"wal-append.run_wall_time_us":1877862}]]}
I20251215 08:59:56.216789 24512 raft_consensus.cc:493] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
W20251215 08:59:56.214319 24543 log.cc:927] Time spent T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20251215 08:59:56.225147 24523 raft_consensus.cc:3060] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 2 FOLLOWER]: Advancing to term 3
I20251215 08:59:56.256191 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:56.256492 21944 leader_election.cc:336] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.523s (SENT)
I20251215 08:59:56.213838 24248 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.249927 24512 raft_consensus.cc:3060] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Advancing to term 10
W20251215 08:59:56.268415 21946 leader_election.cc:336] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447): Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.541s (SENT)
W20251215 08:59:56.260038 24525 log.cc:927] Time spent T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.345s user 0.001s sys 0.001s
W20251215 08:59:56.278046 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 215262us
I20251215 08:59:56.286005 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:56.288653 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:56.290316 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 301563us
I20251215 08:59:56.275113 24393 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:56.293421 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.307734 21979 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:56.248756 24258 consensus_queue.cc:260] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.318722 21378 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:56.298688 24361 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:56.324834 21230 leader_election.cc:336] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.705s (SENT)
I20251215 08:59:56.327804 21230 leader_election.cc:304] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:56.293437 24177 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:56.302662 24475 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:56.332093 21972 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46158 (request call id 701) took 1625 ms. Trace:
W20251215 08:59:56.332429 24546 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P f0824fa5450345719cac22fc6ae3ce2e: Append to log took a long time: real 0.172s user 0.000s sys 0.002s
W20251215 08:59:56.337208 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.332569 21972 rpcz_store.cc:276] 1215 08:59:54.706004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:54.706162 (+ 158us) service_pool.cc:225] Handling call
1215 08:59:55.457462 (+751300us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:55.458328 (+ 866us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:55.458377 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:55.459165 (+ 788us) log.cc:844] Serialized 26 byte log entry
1215 08:59:55.503778 (+ 44613us) raft_consensus.cc:1645] Marking committed up to 7
1215 08:59:55.562787 (+ 59009us) log.cc:844] Serialized 14 byte log entry
1215 08:59:55.563166 (+ 379us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:55.572640 (+ 9474us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:55.684648 (+112008us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b180048ef20. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b06112 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:55.700674 (+ 16026us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:56.068599 (+367925us) raft_consensus.cc:1685] finished
1215 08:59:56.068619 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:56.330831 (+262212us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b540064d7d0. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:56.330951 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32322,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":6481,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":11327,"spinlock_wait_cycles":1723408384,"thread_start_us":34586,"threads_started":1,"wal-append.queue_time_us":53376}
W20251215 08:59:56.349505 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.334182 24421 raft_consensus.cc:2749] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:56.359315 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:56.365496 21259 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:56.368845 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 158087us
W20251215 08:59:56.380009 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:56.384135 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:56.408751 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 56672.9us
I20251215 08:59:56.380232 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:56.408229 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 44379.5us
I20251215 08:59:56.411700 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:56.379369 24258 raft_consensus.cc:3060] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:56.430642 21973 raft_consensus.cc:1240] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:56.443337 24064 consensus_queue.cc:1059] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251215 08:59:56.431509 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:56.440287 21266 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:56.378875 24361 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:56.450021 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.413961 24248 consensus_queue.cc:1059] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20251215 08:59:56.469359 21259 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:56.425561 24357 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.472892 21946 leader_election.cc:304] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:56.478946 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:56.451941 24559 raft_consensus.cc:493] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:56.474241 24500 consensus_meta.cc:287] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.792s user 0.013s sys 0.001s
W20251215 08:59:56.485780 24549 log.cc:927] Time spent T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20251215 08:59:56.474227 24324 raft_consensus.cc:2804] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20251215 08:59:56.478505 24548 log.cc:927] Time spent T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.205s user 0.001s sys 0.001s
I20251215 08:59:56.502285 21332 raft_consensus.cc:1275] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:56.490305 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "beefe15185ac42ef9a20251007f39a56"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:56.503130 21415 raft_consensus.cc:606] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Received request to transfer leadership to TS beefe15185ac42ef9a20251007f39a56
I20251215 08:59:56.503496 21415 raft_consensus.cc:614] T 0c3a05444c334cad8e7562a15e4d68ba P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:56.482555 24085 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:56.493326 24558 raft_consensus.cc:493] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:56.504570 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.480885 24248 raft_consensus.cc:2804] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:56.495118 21339 raft_consensus.cc:1275] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251215 08:59:56.504424 24558 raft_consensus.cc:515] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.494145 24361 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:56.493765 24559 raft_consensus.cc:515] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.510033 24357 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:56.526068 21381 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:56.527505 24361 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:56.505779 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.513228 24248 raft_consensus.cc:697] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:56.516280 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:56.518144 24558 leader_election.cc:290] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.519649 21265 raft_consensus.cc:1240] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:56.519737 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:56.505594 24085 raft_consensus.cc:493] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:56.520653 24554 log.cc:927] Time spent T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20251215 08:59:56.521450 24553 log.cc:927] Time spent T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20251215 08:59:56.521693 24559 leader_election.cc:290] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.522421 24474 raft_consensus.cc:3055] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Stepping down as leader of term 8
I20251215 08:59:56.552698 24248 consensus_queue.cc:237] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.553308 24474 raft_consensus.cc:740] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:56.553192 24559 consensus_queue.cc:1059] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:56.554195 24559 raft_consensus.cc:3055] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 LEADER]: Stepping down as leader of term 9
I20251215 08:59:56.554572 24559 raft_consensus.cc:740] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:56.554315 24474 consensus_queue.cc:260] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.555004 21259 raft_consensus.cc:1275] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20251215 08:59:56.555317 24559 consensus_queue.cc:260] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.555861 24474 raft_consensus.cc:3060] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:56.557277 24559 raft_consensus.cc:3060] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Advancing to term 10
W20251215 08:59:56.559216 24215 consensus_meta.cc:287] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.734s user 0.011s sys 0.000s
I20251215 08:59:56.549501 24558 raft_consensus.cc:493] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:56.513223 21379 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:56.559906 24558 raft_consensus.cc:515] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:56.560721 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 47634.9us
I20251215 08:59:56.561470 21334 raft_consensus.cc:2410] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:56.563314 24475 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:56.563819 24475 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:56.565450 21230 leader_election.cc:336] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.916s (SENT)
I20251215 08:59:56.565829 24475 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.565912 21230 leader_election.cc:304] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:56.532243 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.535264 21337 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:56.535496 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:56.567126 21415 raft_consensus.cc:606] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:56.567664 24479 raft_consensus.cc:2749] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:56.568339 24479 consensus_queue.cc:1048] T 641cd3a624114cf98d427a39378b6da6 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:56.569252 24300 raft_consensus.cc:3055] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 10 LEADER]: Stepping down as leader of term 10
I20251215 08:59:56.569741 24300 raft_consensus.cc:740] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:56.569643 23974 consensus_queue.cc:1048] T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251215 08:59:56.571012 23959 consensus_queue.cc:1048] T 7da883c18eeb4679974472b9b341b5e8 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251215 08:59:56.570811 24300 consensus_queue.cc:260] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.572114 24300 raft_consensus.cc:3060] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Advancing to term 11
I20251215 08:59:56.498441 24324 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:56.573115 24324 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Advancing to term 11
I20251215 08:59:56.548296 21264 raft_consensus.cc:1240] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 9. Current term is 10. Ops: []
I20251215 08:59:56.552452 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:56.546250 24085 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
W20251215 08:59:56.581418 24560 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20251215 08:59:56.581916 24333 consensus_queue.cc:1048] T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:56.583194 21379 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:56.583693 24558 leader_election.cc:290] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.571764 23974 consensus_queue.cc:1059] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251215 08:59:56.586309 21415 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:56.586869 21415 raft_consensus.cc:606] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
W20251215 08:59:56.528831 24555 log.cc:927] Time spent T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.166s user 0.000s sys 0.003s
I20251215 08:59:56.587188 21415 raft_consensus.cc:614] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:56.568476 24482 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.497756 24500 raft_consensus.cc:515] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.606783 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.610585 24150 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:56.610985 21379 leader_election.cc:304] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:56.611261 24489 consensus_meta.cc:287] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.917s user 0.014s sys 0.001s
I20251215 08:59:56.612088 24489 raft_consensus.cc:2749] T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:56.611116 24150 raft_consensus.cc:493] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:56.613416 21413 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
W20251215 08:59:56.644874 21230 leader_election.cc:336] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.078s (SENT)
I20251215 08:59:56.645566 21230 leader_election.cc:304] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:56.648600 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:56.660071 21944 leader_election.cc:336] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.503s (SENT)
I20251215 08:59:56.662992 21413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32818 (request call id 1893) took 1399 ms. Trace:
I20251215 08:59:56.663507 21413 rpcz_store.cc:276] 1215 08:59:55.263378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:55.333728 (+ 70350us) service_pool.cc:225] Handling call
1215 08:59:56.662763 (+1329035us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400293e54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:56.662900 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2912196224}
I20251215 08:59:56.666923 21944 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:56.616580 24150 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:56.675966 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "beefe15185ac42ef9a20251007f39a56"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:56.677544 21336 raft_consensus.cc:1240] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 3. Current term is 4. Ops: []
I20251215 08:59:56.611295 24085 raft_consensus.cc:515] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.677690 24064 consensus_queue.cc:1048] T b7bfa9bddb4b48d996b8b347c898a3b7 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20251215 08:59:56.683630 24500 leader_election.cc:290] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.678201 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.687036 21337 raft_consensus.cc:2410] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:56.671068 24177 raft_consensus.cc:3055] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:56.676894 21413 raft_consensus.cc:606] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Received request to transfer leadership to TS beefe15185ac42ef9a20251007f39a56
I20251215 08:59:56.646857 24349 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20251215 08:59:56.673084 24475 raft_consensus.cc:2749] T 5116dbb730274ac9982610c7f5f8f5cd P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251215 08:59:56.679560 24530 log.cc:927] Time spent T a76bbffe260349a49c7cad7a3de2a9da P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
W20251215 08:59:56.691138 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 67500.7us
I20251215 08:59:56.697075 21259 raft_consensus.cc:1240] T 8d7af00ee6834714a059e045b3c13322 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer 195a56ac72a84018880bfb3bf6e6ef1d for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:56.707177 21337 raft_consensus.cc:1275] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251215 08:59:56.693327 24310 raft_consensus.cc:2749] T 4d1d9a9be078441587cf5da752d59c33 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:56.692000 24177 raft_consensus.cc:740] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:56.732934 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:56.738206 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 50528.7us
I20251215 08:59:56.747372 21337 raft_consensus.cc:2376] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7. Re-sending same reply.
I20251215 08:59:56.691653 24333 consensus_queue.cc:1059] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
W20251215 08:59:56.757450 24389 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.035s sys 0.056s
W20251215 08:59:56.758000 24389 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s user 0.036s sys 0.056s
I20251215 08:59:56.759234 24333 raft_consensus.cc:3055] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Stepping down as leader of term 7
I20251215 08:59:56.759605 24333 raft_consensus.cc:740] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:56.760450 24333 consensus_queue.cc:260] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.762202 24333 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:56.764084 24469 consensus_meta.cc:287] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.607s user 0.008s sys 0.002s
I20251215 08:59:56.763741 24150 raft_consensus.cc:515] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.764504 24469 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.765766 21339 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:56.766535 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:56.774701 21230 leader_election.cc:336] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.128s (SENT)
I20251215 08:59:56.775063 21230 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:56.769994 21337 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 10.
I20251215 08:59:56.765543 24482 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:56.768021 24421 raft_consensus.cc:2749] T 109b80fba7b544e9946db3216c2c7581 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:56.789273 24354 consensus_queue.cc:1059] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20251215 08:59:56.790167 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:56.790385 24523 consensus_meta.cc:287] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.512s user 0.011s sys 0.000s
I20251215 08:59:56.790298 23974 consensus_queue.cc:1048] T 84bdfb60f8fa4b39bc8cbd437a0809f2 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:56.790786 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:56.770845 24475 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:56.798628 21335 raft_consensus.cc:2376] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 6. Re-sending same reply.
I20251215 08:59:56.769752 24258 raft_consensus.cc:493] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader beefe15185ac42ef9a20251007f39a56)
I20251215 08:59:56.777061 24562 raft_consensus.cc:493] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:56.799853 21413 raft_consensus.cc:606] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:56.804911 21413 raft_consensus.cc:614] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:56.783509 24442 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:56.783823 24371 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251215 08:59:56.822505 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 108950us
I20251215 08:59:56.825053 21337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.834230 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:56.834825 21973 raft_consensus.cc:3060] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 5 FOLLOWER]: Advancing to term 7
I20251215 08:59:56.829865 23846 consensus_queue.cc:1048] T 64b5a2ef4843492085aca418ad649914 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20251215 08:59:56.835901 24310 consensus_queue.cc:1059] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251215 08:59:56.836138 24489 consensus_queue.cc:1048] T 109b80fba7b544e9946db3216c2c7581 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:56.836776 24310 raft_consensus.cc:3055] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Stepping down as leader of term 9
I20251215 08:59:56.837167 24310 raft_consensus.cc:740] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
W20251215 08:59:56.838837 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.832899 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:59:56.841759 20109 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251215 08:59:56.801383 24523 raft_consensus.cc:515] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.843719 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.833199 24085 leader_election.cc:290] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:56.849715 21972 raft_consensus.cc:1240] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:56.807126 24475 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.850154 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:56.837455 24569 raft_consensus.cc:493] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:56.856688 21979 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:56.856770 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:56.840342 24512 consensus_meta.cc:287] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.554s user 0.011s sys 0.001s
I20251215 08:59:56.831223 24150 leader_election.cc:290] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:56.851047 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 82645us
I20251215 08:59:56.808250 24258 raft_consensus.cc:515] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.853292 21339 raft_consensus.cc:2393] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 6: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:56.839931 24514 consensus_meta.cc:287] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.818s user 0.005s sys 0.008s
I20251215 08:59:56.869727 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:56.870237 24523 leader_election.cc:290] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 3 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:56.871621 21230 leader_election.cc:336] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.609s (SENT)
W20251215 08:59:56.872586 21230 leader_election.cc:336] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.086s (SENT)
I20251215 08:59:56.874244 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.875083 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.875563 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.875859 21332 raft_consensus.cc:2368] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:56.876312 21333 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:56.877112 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.877381 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.877982 21334 raft_consensus.cc:2376] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 7. Re-sending same reply.
I20251215 08:59:56.878865 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.879114 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.879534 21334 raft_consensus.cc:2368] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:56.879629 21341 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Advancing to term 11
I20251215 08:59:56.880256 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.876700 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.881193 24310 consensus_queue.cc:260] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.881740 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.865830 24569 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.806118 24324 raft_consensus.cc:515] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.810992 24562 raft_consensus.cc:515] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.867317 24482 consensus_queue.cc:1048] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251215 08:59:56.882858 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.890607 21333 raft_consensus.cc:2410] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20251215 08:59:56.884991 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.890887 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:56.891698 21259 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
W20251215 08:59:56.893643 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:56.889561 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.894481 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.895484 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:56.896164 21979 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 3: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:56.898015 24562 leader_election.cc:290] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.900195 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.901382 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.835225 24442 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:56.902688 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.903329 21333 raft_consensus.cc:2368] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7.
I20251215 08:59:56.883983 24512 raft_consensus.cc:515] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:56.904120 24442 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:56.904780 24310 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:56.903198 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.904134 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.900895 21340 raft_consensus.cc:2368] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 2. Current term is 3.
I20251215 08:59:56.904475 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.912695 21339 raft_consensus.cc:2393] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:56.913924 21257 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:56.916076 21333 raft_consensus.cc:2368] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 6.
I20251215 08:59:56.916453 21335 raft_consensus.cc:2393] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:56.905196 24566 consensus_queue.cc:1059] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20251215 08:59:56.923359 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.925897 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:56.929925 21260 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:56.933609 21259 raft_consensus.cc:2468] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:56.926666 21337 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 10: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:56.906603 24520 consensus_meta.cc:287] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.852s user 0.006s sys 0.006s
W20251215 08:59:56.927260 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 91239.3us
I20251215 08:59:56.935609 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.936744 21338 raft_consensus.cc:1240] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 9. Current term is 10. Ops: []
W20251215 08:59:56.940461 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.938875 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.938875 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.942353 21339 raft_consensus.cc:2410] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:56.943128 21381 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:56.923681 24469 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.953990 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.955128 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.956066 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.957265 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.956905 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.959787 21332 raft_consensus.cc:2468] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
W20251215 08:59:56.959949 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:56.959816 21340 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:56.960961 24579 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:56.961465 24579 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:56.961812 24579 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:56.965288 21332 raft_consensus.cc:1240] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 1. Current term is 3. Ops: []
I20251215 08:59:56.959180 21946 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:56.972889 24442 consensus_queue.cc:1048] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:56.977739 24324 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 11 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:56.982097 24475 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.983033 24479 consensus_queue.cc:1059] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251215 08:59:56.984138 24479 raft_consensus.cc:3055] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Stepping down as leader of term 3
I20251215 08:59:56.984493 24479 raft_consensus.cc:740] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:56.988476 21341 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 11.
I20251215 08:59:56.986409 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:56.997077 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:56.984925 24258 leader_election.cc:290] T 0922c97d6dce49f0a3b98d200f4f651e P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:56.904171 24177 consensus_queue.cc:260] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.002775 21339 raft_consensus.cc:2410] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251215 08:59:57.004101 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 51140.2us
W20251215 08:59:57.005034 21230 leader_election.cc:336] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.565s (SENT)
I20251215 08:59:57.005506 21230 leader_election.cc:304] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:56.942484 24520 raft_consensus.cc:515] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.010393 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.014722 21412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:57.017876 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.018855 21339 raft_consensus.cc:2393] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:57.014180 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:57.023030 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.023161 21412 raft_consensus.cc:3060] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:57.023641 21341 raft_consensus.cc:2468] T 07b6839b646d44fbac5ae9e9e2a65daa P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:57.024691 24475 raft_consensus.cc:2749] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:57.034610 24520 leader_election.cc:290] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:56.958293 24248 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
W20251215 08:59:57.057103 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.008172 24236 raft_consensus.cc:2804] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:57.059723 21336 raft_consensus.cc:2368] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7.
W20251215 08:59:57.036888 21944 leader_election.cc:336] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.541s (SENT)
I20251215 08:59:57.063053 21944 leader_election.cc:304] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:57.023828 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.073750 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.064431 21973 raft_consensus.cc:2468] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:57.074657 21335 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:57.081312 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.085750 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.088052 21339 raft_consensus.cc:2410] T 01ad5646d4d04a6a996ce407925b8650 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251215 08:59:57.104147 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.106132 21341 raft_consensus.cc:2468] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9.
I20251215 08:59:57.106216 21335 raft_consensus.cc:2468] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:57.118053 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:57.121287 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 116310us
I20251215 08:59:56.996649 24479 consensus_queue.cc:260] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:57.122977 21230 leader_election.cc:336] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.854s (SENT)
I20251215 08:59:57.127027 21230 leader_election.cc:304] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:57.125224 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109b80fba7b544e9946db3216c2c7581" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.128840 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.129179 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.130043 21339 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:57.132225 21335 raft_consensus.cc:2393] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:57.089862 24512 leader_election.cc:290] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:57.140184 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.119735 24569 leader_election.cc:290] T 15d0087ada21421eacebc646b271e145 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:57.139318 24475 raft_consensus.cc:2749] T 1db12563817b44429a055b7dcdd066c3 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:57.168746 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.169100 21412 raft_consensus.cc:2468] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:57.169121 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5116dbb730274ac9982610c7f5f8f5cd" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.171515 21339 raft_consensus.cc:2468] T 5116dbb730274ac9982610c7f5f8f5cd P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:57.176221 24479 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:57.181404 24274 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:57.182004 24274 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.182694 21415 raft_consensus.cc:2468] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:57.182907 24150 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:57.183470 24150 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:57.184048 24274 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:57.185307 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.186038 21341 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:57.186146 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:57.186781 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.187054 21338 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:57.187485 24310 raft_consensus.cc:2749] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:57.188118 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.136235 24177 raft_consensus.cc:3060] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:57.174086 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9582bb1875204089bca2e38a3ff426b3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:57.187493 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.191917 21335 raft_consensus.cc:3060] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:57.188817 21338 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:57.193787 21341 raft_consensus.cc:2393] T 1db12563817b44429a055b7dcdd066c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 10: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:57.194826 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:57.195904 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 72988.9us
I20251215 08:59:57.207007 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b5a2ef4843492085aca418ad649914" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.207648 21334 raft_consensus.cc:2468] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:57.140267 24579 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:57.211241 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.211410 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.212244 21341 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:57.195855 24236 raft_consensus.cc:697] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:57.315368 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76bbffe260349a49c7cad7a3de2a9da" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.245115 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.317803 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:57.329677 21266 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:57.244204 21340 raft_consensus.cc:1240] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 7. Ops: []
W20251215 08:59:57.215951 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 210569us
W20251215 08:59:57.306396 21230 leader_election.cc:336] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.791s (SENT)
I20251215 08:59:57.331043 21334 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:57.239089 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:57.327724 21341 raft_consensus.cc:2468] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:57.241133 24150 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:57.332736 21230 leader_election.cc:304] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:57.328577 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:57.292913 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.335323 21412 raft_consensus.cc:1275] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251215 08:59:57.335602 21338 raft_consensus.cc:2410] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251215 08:59:57.316460 24583 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20251215 08:59:57.331702 21263 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:57.336925 21339 raft_consensus.cc:2393] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:57.340174 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.342525 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.344767 21338 raft_consensus.cc:2410] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:57.343617 21381 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:57.287354 24592 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:57.351594 21229 leader_election.cc:304] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:57.352402 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:57.353909 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da883c18eeb4679974472b9b341b5e8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.275734 24512 raft_consensus.cc:493] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:57.299786 24579 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:57.209439 24482 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.147s
I20251215 08:59:57.365363 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.362752 21334 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:57.396596 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.398690 21341 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:57.406369 24569 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:57.412402 24569 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.414412 24569 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:57.415177 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:57.415804 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 113752us
W20251215 08:59:57.425114 24583 log.cc:927] Time spent T 5d93e1c6de78493c926c5be7096cfb31 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
W20251215 08:59:57.426826 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.427376 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:57.435818 21979 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:57.428052 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:57.429222 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:57.430281 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.437143 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:57.408640 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c3a05444c334cad8e7562a15e4d68ba" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:57.365072 24371 raft_consensus.cc:2749] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:57.417326 24236 consensus_queue.cc:237] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.434249 21334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:57.435022 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 244540us
I20251215 08:59:57.438022 21338 raft_consensus.cc:2468] T 7da883c18eeb4679974472b9b341b5e8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:57.435046 21341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.368041 24475 raft_consensus.cc:2804] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:57.442046 21334 raft_consensus.cc:2410] T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:57.442804 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:57.443459 21975 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:57.440883 21230 leader_election.cc:336] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.874s (SENT)
I20251215 08:59:57.444653 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:57.435485 24333 consensus_meta.cc:287] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.673s user 0.012s sys 0.000s
I20251215 08:59:57.364908 24489 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:57.447130 21339 raft_consensus.cc:2410] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251215 08:59:57.447180 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 410293us
I20251215 08:59:57.446689 21946 leader_election.cc:304] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:57.448817 21335 raft_consensus.cc:2468] T 9582bb1875204089bca2e38a3ff426b3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
W20251215 08:59:57.452046 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.452666 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:57.452914 21338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.453588 21338 raft_consensus.cc:1813] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Handling vote request from an unknown peer beefe15185ac42ef9a20251007f39a56
I20251215 08:59:57.453994 21338 raft_consensus.cc:2410] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251215 08:59:57.455171 21334 raft_consensus.cc:1240] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:57.455176 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d7af00ee6834714a059e045b3c13322" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:57.457106 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.367619 24592 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.458112 21339 raft_consensus.cc:1275] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20251215 08:59:57.456878 21232 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:57.460850 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:57.461804 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:57.448019 24489 raft_consensus.cc:493] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:57.458940 21339 pending_rounds.cc:85] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 5
I20251215 08:59:57.469496 21339 pending_rounds.cc:107] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20251215 08:59:57.469578 21266 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:57.470019 21339 raft_consensus.cc:2889] T e80fc34b30d346baa0dfec4df957cd9f P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:57.450879 24248 raft_consensus.cc:2749] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:57.478224 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:57.479085 21413 raft_consensus.cc:2393] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 8: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:57.479543 21338 raft_consensus.cc:1275] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: Refusing update from remote peer 195a56ac72a84018880bfb3bf6e6ef1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20251215 08:59:57.481036 21338 pending_rounds.cc:85] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 4
I20251215 08:59:57.481564 21338 pending_rounds.cc:107] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20251215 08:59:57.470881 21381 leader_election.cc:304] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:57.482124 21338 raft_consensus.cc:2889] T b83855234bda47a1b06c3f932728766b P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 08:59:57.487468 21414 raft_consensus.cc:2468] T 8d7af00ee6834714a059e045b3c13322 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:57.442317 24512 raft_consensus.cc:515] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:57.493089 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 52225.5us
I20251215 08:59:57.455067 24475 raft_consensus.cc:697] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [term 7 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:57.495896 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 55461.1us
I20251215 08:59:57.495383 24064 raft_consensus.cc:2804] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:57.496747 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:57.499583 21414 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:57.468390 24489 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:57.492439 24371 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:57.541500 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067a0581bd0244539c1eca2c554e4619" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.502719 24064 raft_consensus.cc:697] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
W20251215 08:59:57.544459 24605 log.cc:927] Time spent T 07b6839b646d44fbac5ae9e9e2a65daa P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20251215 08:59:57.544203 24064 consensus_queue.cc:237] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:57.545189 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 89121.1us
I20251215 08:59:57.547480 21341 raft_consensus.cc:2468] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:57.547981 21333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0351e3362cb742aa87ffc4bf31ceea2c" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.578996 21333 raft_consensus.cc:2410] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:57.583025 21336 raft_consensus.cc:2410] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251215 08:59:57.548892 24371 raft_consensus.cc:493] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:57.596487 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:57.699116 21230 leader_election.cc:336] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.003s (SENT)
I20251215 08:59:57.700878 21229 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:57.594698 24562 raft_consensus.cc:3055] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 LEADER]: Stepping down as leader of term 8
W20251215 08:59:57.695398 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.717940 24602 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:57.695749 24371 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:57.726261 24592 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:57.725451 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:57.725503 24475 consensus_queue.cc:237] T 3d471c9f4eb74ea3a20cec86631e4ab0 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.749910 24602 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:57.752125 21232 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.163.67:37447 timed out after 1.511s (SENT) [suppressed 66 similar messages]
I20251215 08:59:57.711056 24562 raft_consensus.cc:740] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0a2fcb89589c401d8efc8d86da4ba0f4, State: Running, Role: LEADER
I20251215 08:59:57.783389 24602 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:57.892423 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:57.930819 24562 consensus_queue.cc:260] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:57.932575 24562 raft_consensus.cc:3060] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:57.861143 24512 leader_election.cc:290] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:57.976500 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:58.001163 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.087713 24592 raft_consensus.cc:493] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:58.088478 24592 raft_consensus.cc:515] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.090865 24592 leader_election.cc:290] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.122159 20109 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251215 08:59:58.076826 24482 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:58.134310 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.135157 21340 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:58.136207 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:58.137084 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 437985us
W20251215 08:59:58.116101 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.139109 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:58.116371 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.140148 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
W20251215 08:59:58.141088 21944 leader_election.cc:336] T 458cec829de44303951b997be68accde P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.656s (SENT)
W20251215 08:59:58.140163 21257 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.027s sys 0.132s
W20251215 08:59:58.149341 21257 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.650s user 0.027s sys 0.132s
I20251215 08:59:58.146898 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60989b82e8354098aac2066e31697953" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:58.143447 21230 leader_election.cc:336] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.993s (SENT)
I20251215 08:59:58.152606 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.154359 21335 raft_consensus.cc:2393] T 3d471c9f4eb74ea3a20cec86631e4ab0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:58.154532 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.136734 21340 raft_consensus.cc:2468] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 5.
I20251215 08:59:58.155503 21414 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 9: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:58.159951 21230 leader_election.cc:304] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:58.162408 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:58.150084 24064 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.000s sys 0.025s
I20251215 08:59:58.166049 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:58.169150 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:58.170588 21406 raft_consensus.cc:2376] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8. Re-sending same reply.
I20251215 08:59:58.170819 21407 raft_consensus.cc:2376] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 11. Re-sending same reply.
I20251215 08:59:58.172792 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:58.176322 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.173738 21340 raft_consensus.cc:2468] T 9b36e018bdd140648e80e809a47269c8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:58.185922 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:58.170305 24064 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.000s sys 0.026s
I20251215 08:59:58.146862 24482 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.205310 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69415d618114174b070bbc19a876119" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.206307 21335 raft_consensus.cc:2468] T d69415d618114174b070bbc19a876119 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:58.206627 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:58.213315 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.223330 21339 raft_consensus.cc:2393] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 3: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
W20251215 08:59:58.217625 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 76585.5us
I20251215 08:59:58.225128 21340 raft_consensus.cc:2468] T ba393161b7454821b7584a44b64bf6b2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:58.226329 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:58.216065 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.228227 21335 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:58.229825 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
W20251215 08:59:58.230110 21944 leader_election.cc:336] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.826s (SENT)
I20251215 08:59:58.230582 21335 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:58.228370 21407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:58.231194 21407 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:58.231446 21335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124c9efc80174ca585aaee7e0f8a12f0" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.232403 21406 raft_consensus.cc:1240] T a76bbffe260349a49c7cad7a3de2a9da P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:58.216022 24248 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.234833 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6053bc9278744c6b93f60f788aa7b06e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.234964 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.235411 21340 raft_consensus.cc:2468] T 6053bc9278744c6b93f60f788aa7b06e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:58.235474 21339 raft_consensus.cc:2468] T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 7.
I20251215 08:59:58.225733 24064 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.240345 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.243152 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.243620 21339 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 7: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:58.243767 21340 raft_consensus.cc:2393] T caf0c0cd96174b4889e367b1f98d617e P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 4: Already voted for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in this term.
I20251215 08:59:58.242170 24482 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.245292 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1899) took 1343 ms. Trace:
I20251215 08:59:58.246236 21332 rpcz_store.cc:276] 1215 08:59:56.899162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:56.960488 (+ 61326us) service_pool.cc:225] Handling call
1215 08:59:56.965100 (+ 4612us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:56.965735 (+ 635us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:58.242418 (+1276683us) spinlock_profiling.cc:243] Waited 4.33 ms on lock 0x7b5400047c50. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb352d 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:58.242557 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9963904}
I20251215 08:59:58.247190 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:58.250442 21944 leader_election.cc:336] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.850s (SENT)
I20251215 08:59:58.250757 21944 leader_election.cc:304] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:58.254082 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.255913 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.256374 21340 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 6.
I20251215 08:59:58.238544 24248 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.261168 21408 raft_consensus.cc:1240] T b3680af750ef47df88bf5be33eb18fc7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer 0a2fcb89589c401d8efc8d86da4ba0f4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20251215 08:59:58.248311 24482 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:58.259591 21944 leader_election.cc:336] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.989s (SENT)
I20251215 08:59:58.264403 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:58.264710 21340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641cd3a624114cf98d427a39378b6da6" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.265659 21340 raft_consensus.cc:2468] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 6.
I20251215 08:59:58.268823 24482 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.242683 24064 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.269124 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.281558 21339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:58.259945 24569 raft_consensus.cc:493] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:58.284191 24569 raft_consensus.cc:515] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.286304 24569 leader_election.cc:290] T ac8397b87ffc453b8f468eee77a8e410 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:58.258888 24512 raft_consensus.cc:493] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:58.287427 24512 raft_consensus.cc:515] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.289518 24512 leader_election.cc:290] T 0cdd8fe5ea144c7897b7cf7eaf77c19a P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:58.286809 24569 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:58.290922 24569 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:58.292694 24371 consensus_meta.cc:287] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 0.509s user 0.010s sys 0.000s
I20251215 08:59:58.293617 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2012) took 1230 ms. Trace:
W20251215 08:59:58.294282 21381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 154859us
I20251215 08:59:58.258280 24236 raft_consensus.cc:2749] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:58.300534 21257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50710 (request call id 2275) took 1059 ms. Trace:
I20251215 08:59:58.302029 24475 raft_consensus.cc:2749] T 7da883c18eeb4679974472b9b341b5e8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:58.315490 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.318732 21408 raft_consensus.cc:2393] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 9: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
W20251215 08:59:58.309207 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 175533us
I20251215 08:59:58.306500 21257 rpcz_store.cc:276] 1215 08:59:57.241149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:57.241380 (+ 231us) service_pool.cc:225] Handling call
1215 08:59:57.241781 (+ 401us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:57.242112 (+ 331us) raft_consensus.cc:1534] Early marking committed up to index 5
1215 08:59:58.149701 (+907589us) log.cc:844] Serialized 14 byte log entry
1215 08:59:58.300167 (+150466us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:58.300223 (+ 56us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:58.300281 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:58.300344 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:58.300479 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":240004,"prepare.queue_time_us":268021,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":800630,"threads_started":2,"wal-append.queue_time_us":150440}
I20251215 08:59:58.294718 21334 rpcz_store.cc:276] 1215 08:59:57.063190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:57.478919 (+415729us) service_pool.cc:225] Handling call
1215 08:59:57.479250 (+ 331us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:57.483868 (+ 4618us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:57.483894 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:57.489830 (+ 5936us) log.cc:844] Serialized 26 byte log entry
1215 08:59:57.696833 (+207003us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:57.696875 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:57.696948 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:58.293355 (+596407us) raft_consensus.cc:1685] finished
1215 08:59:58.293382 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:58.293547 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4400,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4978,"thread_start_us":86086,"threads_started":1,"wal-append.queue_time_us":667832}
I20251215 08:59:58.286425 24248 leader_election.cc:290] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.332715 21407 raft_consensus.cc:2410] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251215 08:59:58.296149 24482 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:58.342967 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:58.343086 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 87898.5us
I20251215 08:59:58.349972 21414 raft_consensus.cc:1240] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 7. Current term is 8. Ops: []
I20251215 08:59:58.350641 21413 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 8. Current term is 9. Ops: []
I20251215 08:59:58.350417 21232 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:58.351698 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.352349 21414 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 8.
I20251215 08:59:58.356725 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.357465 21232 leader_election.cc:304] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:58.362118 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:58.367909 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:58.368839 21414 raft_consensus.cc:2376] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9. Re-sending same reply.
I20251215 08:59:58.370121 21379 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:58.367223 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8b3d79d7a014e568e385110044efb63" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.304457 24371 raft_consensus.cc:515] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.373543 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f800ccb361734ac18102a3502ee639d8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.401870 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.405493 21406 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251215 08:59:58.406656 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:58.407406 21979 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:58.408437 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:58.413461 21975 raft_consensus.cc:2410] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20251215 08:59:58.347506 24569 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.420002 24371 leader_election.cc:290] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 election: Requested vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:58.422493 24357 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.423161 24219 raft_consensus.cc:493] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251215 08:59:58.429212 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 104116us
W20251215 08:59:58.442121 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.373520 24248 raft_consensus.cc:493] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.373559 24236 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.452345 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:58.453140 21973 raft_consensus.cc:2393] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 in current term 9: Already voted for candidate f0824fa5450345719cac22fc6ae3ce2e in this term.
W20251215 08:59:58.469214 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:58.442663 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:58.441797 24591 log.cc:927] Time spent T 00000000000000000000000000000000 P 1b8aeedd216e479da2a4e44a3dd8b23f: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20251215 08:59:58.479892 21337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47830 (request call id 370) took 1651 ms (client timeout 1850 ms). Trace:
W20251215 08:59:58.486655 21337 rpcz_store.cc:269] 1215 08:59:56.798225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:56.807901 (+ 9676us) service_pool.cc:225] Handling call
1215 08:59:58.448975 (+1641074us) spinlock_profiling.cc:243] Waited 86.3 ms on lock 0x7b5400153e54. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:58.449202 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198500608}
I20251215 08:59:58.483841 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:58.488232 21979 raft_consensus.cc:3060] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Advancing to term 11
I20251215 08:59:58.498811 21415 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.499226 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.445398 24627 raft_consensus.cc:493] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.503034 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.417743 24624 consensus_queue.cc:1048] T 0351e3362cb742aa87ffc4bf31ceea2c P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20251215 08:59:58.506623 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:58.507628 21414 raft_consensus.cc:2410] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20251215 08:59:58.506024 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 148678us
I20251215 08:59:58.508693 21232 leader_election.cc:304] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:58.495090 24562 consensus_meta.cc:287] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4: Time spent flushing consensus metadata: real 0.556s user 0.012s sys 0.000s
I20251215 08:59:58.453404 24482 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:58.499887 21406 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
W20251215 08:59:58.510890 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 367453us
I20251215 08:59:58.511770 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.489691 24371 raft_consensus.cc:2749] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20251215 08:59:58.515653 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 145653us
I20251215 08:59:58.523970 21335 raft_consensus.cc:2368] T 124c9efc80174ca585aaee7e0f8a12f0 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate beefe15185ac42ef9a20251007f39a56 for earlier term 8. Current term is 9.
I20251215 08:59:58.524091 21232 leader_election.cc:304] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:58.525208 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.525825 21337 raft_consensus.cc:1240] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20251215 08:59:58.523324 21379 leader_election.cc:304] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:58.526443 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1937) took 1379 ms. Trace:
I20251215 08:59:58.526803 21337 rpcz_store.cc:276] 1215 08:59:57.146774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.510516 (+1363742us) service_pool.cc:225] Handling call
1215 08:59:58.522770 (+ 12254us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:58.526186 (+ 3416us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:58.526389 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:58.527539 21266 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:58.530575 21946 leader_election.cc:304] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:58.445720 24357 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.542812 24482 raft_consensus.cc:2749] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 08:59:58.543551 24357 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:58.545213 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.545897 24064 leader_election.cc:290] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:58.532581 21379 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:58.548669 21379 leader_election.cc:304] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters: f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:58.504832 23959 raft_consensus.cc:2804] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:58.517108 24618 raft_consensus.cc:493] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.533210 24629 raft_consensus.cc:2749] T 0c3a05444c334cad8e7562a15e4d68ba P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251215 08:59:58.553962 21944 leader_election.cc:336] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.185s (SENT)
I20251215 08:59:58.554605 21944 leader_election.cc:304] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:58.538996 24085 raft_consensus.cc:3055] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Stepping down as leader of term 6
I20251215 08:59:58.539243 21979 raft_consensus.cc:2468] T 15d0087ada21421eacebc646b271e145 P f0824fa5450345719cac22fc6ae3ce2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 11.
I20251215 08:59:58.540405 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:58.541046 24620 raft_consensus.cc:2749] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251215 08:59:58.449965 24219 raft_consensus.cc:515] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.476516 24248 raft_consensus.cc:515] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } attrs { replace: true } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.541379 21261 raft_consensus.cc:1275] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251215 08:59:58.510231 24627 raft_consensus.cc:515] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.555259 21258 raft_consensus.cc:1275] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Refusing update from remote peer f0824fa5450345719cac22fc6ae3ce2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251215 08:59:58.479547 24236 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.556484 21406 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.600504 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79790fd760ff4ae1b3879326decc531a" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:58.603994 21257 raft_consensus.cc:2393] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0824fa5450345719cac22fc6ae3ce2e in current term 9: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:58.602437 21413 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2c10c3de775846868774c98cc0a75094"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
W20251215 08:59:58.649221 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.605530 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
W20251215 08:59:58.607762 21230 leader_election.cc:336] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.695s (SENT)
I20251215 08:59:58.660708 21413 raft_consensus.cc:606] T 2c10c3de775846868774c98cc0a75094 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:58.663395 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:58.556195 24357 raft_consensus.cc:493] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:58.592449 21379 leader_election.cc:304] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, beefe15185ac42ef9a20251007f39a56
W20251215 08:59:58.601166 21943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.669355 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.664106 21260 raft_consensus.cc:2468] T 3cf7b0165c00477b9c1b94c6a89052a6 P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 10.
I20251215 08:59:58.557365 21946 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:58.671726 21337 raft_consensus.cc:1275] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251215 08:59:58.574795 21232 leader_election.cc:304] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56; no voters:
I20251215 08:59:58.599529 24064 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:58.672307 21337 pending_rounds.cc:85] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting all ops after (but not including) 2
I20251215 08:59:58.601071 24085 raft_consensus.cc:740] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:58.662083 21230 leader_election.cc:304] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:58.604710 24592 raft_consensus.cc:2749] T 7edcdb5749ae4e59ab17648fe97390e2 P f0824fa5450345719cac22fc6ae3ce2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:58.600544 24594 raft_consensus.cc:2804] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election won for term 8
W20251215 08:59:58.674062 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 81720.2us
I20251215 08:59:58.682118 21337 pending_rounds.cc:107] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251215 08:59:58.717789 21337 raft_consensus.cc:2889] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251215 08:59:58.682307 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:58.683471 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 126203us
W20251215 08:59:58.681200 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 127282us
I20251215 08:59:58.685905 21413 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
W20251215 08:59:58.693782 24599 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 1.252s user 0.000s sys 0.032s
I20251215 08:59:58.671268 24562 raft_consensus.cc:493] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.551913 23959 raft_consensus.cc:493] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:58.702811 21408 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3cf7b0165c00477b9c1b94c6a89052a6"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "beefe15185ac42ef9a20251007f39a56"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:58.757292 21408 raft_consensus.cc:606] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Received request to transfer leadership to TS beefe15185ac42ef9a20251007f39a56
I20251215 08:59:58.757741 21408 raft_consensus.cc:614] T 3cf7b0165c00477b9c1b94c6a89052a6 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251215 08:59:58.684608 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.671237 24620 raft_consensus.cc:2804] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader election won for term 8
W20251215 08:59:58.737965 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:58.839732 21229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 67366.6us
W20251215 08:59:58.756398 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 148652us
I20251215 08:59:58.599334 24618 raft_consensus.cc:515] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:58.846724 21307 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.734602 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac8397b87ffc453b8f468eee77a8e410" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:58.755074 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cdd8fe5ea144c7897b7cf7eaf77c19a" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:58.757225 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 205747us
I20251215 08:59:58.851732 21232 leader_election.cc:304] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:58.742312 21414 raft_consensus.cc:2468] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 4.
W20251215 08:59:58.839289 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.735474 21304 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:58.765628 24625 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 11 FOLLOWER]: Leader election won for term 11
I20251215 08:59:58.675979 24219 leader_election.cc:290] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:58.680124 23974 consensus_queue.cc:1048] T b83855234bda47a1b06c3f932728766b P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.165s
I20251215 08:59:58.673848 24357 raft_consensus.cc:515] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:58.869055 21230 leader_election.cc:336] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 3.345s (SENT)
I20251215 08:59:58.876148 21230 leader_election.cc:304] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:58.861660 21304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.68:45831, user_credentials={real_user=slave}} blocked reactor thread for 126258us
I20251215 08:59:58.733994 24631 raft_consensus.cc:2749] T 0351e3362cb742aa87ffc4bf31ceea2c P 195a56ac72a84018880bfb3bf6e6ef1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:58.789872 23959 raft_consensus.cc:3060] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:58.873551 21408 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
I20251215 08:59:58.878119 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.885830 21408 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 FOLLOWER]: Advancing to term 7
W20251215 08:59:58.885838 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 152653us
I20251215 08:59:58.844619 24371 raft_consensus.cc:493] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:58.886193 21414 raft_consensus.cc:2393] T 0922c97d6dce49f0a3b98d200f4f651e P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 8: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:58.845985 24629 raft_consensus.cc:2749] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251215 08:59:58.887392 21307 connection.cc:232] RPC call timeout handler was delayed by -1.7127952629999754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
W20251215 08:59:58.881028 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 142449us
W20251215 08:59:58.887902 21307 connection.cc:232] RPC call timeout handler was delayed by -1.6380952629999683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17410000000000003s for a response
W20251215 08:59:58.888269 21307 connection.cc:232] RPC call timeout handler was delayed by -1.2645952630000465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20251215 08:59:58.888530 21307 connection.cc:232] RPC call timeout handler was delayed by -1.1826952630000278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2247s for a response
I20251215 08:59:58.846240 24633 raft_consensus.cc:2804] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251215 08:59:58.849164 24482 raft_consensus.cc:493] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:58.850144 24624 raft_consensus.cc:493] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:58.894675 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.773903 24601 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.124s
I20251215 08:59:58.870507 24626 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251215 08:59:58.775195 24562 raft_consensus.cc:515] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.902379 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1926) took 1874 ms. Trace:
I20251215 08:59:58.911746 21336 rpcz_store.cc:276] 1215 08:59:57.027727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:57.540061 (+512334us) service_pool.cc:225] Handling call
1215 08:59:58.902112 (+1362051us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b540009c9d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:58.902302 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92812032}
I20251215 08:59:58.913162 24594 raft_consensus.cc:697] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [term 8 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:58.914290 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.916149 21336 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 8.
I20251215 08:59:58.920006 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15d0087ada21421eacebc646b271e145" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:58.920936 21414 raft_consensus.cc:2393] T 15d0087ada21421eacebc646b271e145 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 11: Already voted for candidate 195a56ac72a84018880bfb3bf6e6ef1d in this term.
I20251215 08:59:58.877506 24625 raft_consensus.cc:697] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [term 11 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:58.879758 24639 raft_consensus.cc:493] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:58.914532 24594 consensus_queue.cc:237] T b3680af750ef47df88bf5be33eb18fc7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.929750 21379 leader_election.cc:304] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:58.899891 24333 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.892053 24621 raft_consensus.cc:3060] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:58.898430 24633 raft_consensus.cc:493] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:58.891760 24569 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:58.949494 21307 leader_election.cc:304] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:58.891382 24512 raft_consensus.cc:493] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
W20251215 08:59:58.950376 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:58.922688 24626 raft_consensus.cc:493] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:58.958442 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.65:46193, user_credentials={real_user=slave}} blocked reactor thread for 57242.6us
W20251215 08:59:58.958261 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 69786.9us
I20251215 08:59:58.897642 24371 raft_consensus.cc:515] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.965684 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.957897 24649 raft_consensus.cc:2804] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251215 08:59:58.968319 24627 leader_election.cc:290] T 0351e3362cb742aa87ffc4bf31ceea2c P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.968410 24649 raft_consensus.cc:493] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:58.968984 24248 leader_election.cc:290] T 5d93e1c6de78493c926c5be7096cfb31 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.910931 24624 raft_consensus.cc:515] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.956136 24333 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.974404 24371 leader_election.cc:290] T 067a0581bd0244539c1eca2c554e4619 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.974576 24624 leader_election.cc:290] T b83855234bda47a1b06c3f932728766b P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:58.974928 24333 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:58.961519 24646 consensus_queue.cc:1048] T f800ccb361734ac18102a3502ee639d8 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
W20251215 08:59:58.975618 24489 consensus_meta.cc:287] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 1.435s user 0.012s sys 0.002s
W20251215 08:59:58.976078 24602 consensus_meta.cc:287] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56: Time spent flushing consensus metadata: real 1.117s user 0.010s sys 0.001s
I20251215 08:59:58.976035 24489 raft_consensus.cc:515] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.930752 24629 raft_consensus.cc:2749] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 08:59:58.979657 21408 raft_consensus.cc:2468] T f4ed801d85bc47d7ac87033f0799bad2 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 7.
I20251215 08:59:58.911144 24482 raft_consensus.cc:515] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:58.977622 24592 consensus_queue.cc:1048] T ba393161b7454821b7584a44b64bf6b2 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
W20251215 08:59:58.977119 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 108140us
I20251215 08:59:58.954172 24633 raft_consensus.cc:3060] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:58.979198 21414 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4d1d9a9be078441587cf5da752d59c33"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:58.977535 21336 raft_consensus.cc:2468] T 88823d4c370d4c6a96a6ba54418638eb P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 9.
I20251215 08:59:58.968829 24649 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:58.989306 21379 leader_election.cc:304] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:58.991859 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.992735 21336 raft_consensus.cc:2410] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251215 08:59:58.959272 24569 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.993667 21336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:58.994205 21336 raft_consensus.cc:2468] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 11.
I20251215 08:59:58.995283 24569 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:58.996122 24248 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251215 08:59:58.961982 24626 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:59.001174 24333 raft_consensus.cc:2804] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251215 08:59:59.001739 24333 raft_consensus.cc:493] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251215 08:59:59.001932 24635 log.cc:927] Time spent T f14cfff8c5a44ad78299ada0ca53baa9 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20251215 08:59:59.002233 24333 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Advancing to term 10
I20251215 08:59:59.006327 24601 consensus_queue.cc:1048] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 08:59:58.988095 21414 raft_consensus.cc:606] T 4d1d9a9be078441587cf5da752d59c33 P 195a56ac72a84018880bfb3bf6e6ef1d [term 6 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
W20251215 08:59:59.008100 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:59.009863 21379 leader_election.cc:304] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
I20251215 08:59:59.013981 21412 raft_consensus.cc:1240] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer beefe15185ac42ef9a20251007f39a56 for earlier term 5. Current term is 7. Ops: []
I20251215 08:59:58.966396 24620 raft_consensus.cc:697] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Becoming Leader. State: Replica: 195a56ac72a84018880bfb3bf6e6ef1d, State: Running, Role: LEADER
I20251215 08:59:59.031023 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e724e065b349faafa05321d2582972" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
I20251215 08:59:59.031610 21979 raft_consensus.cc:3060] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:59.032712 24646 consensus_queue.cc:1048] T 124c9efc80174ca585aaee7e0f8a12f0 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:59.036566 21414 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" is_pre_election: true
I20251215 08:59:59.023247 23974 raft_consensus.cc:2804] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20251215 08:59:59.037168 21414 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0824fa5450345719cac22fc6ae3ce2e in term 9.
I20251215 08:59:59.037369 23974 raft_consensus.cc:493] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:59.037506 24357 leader_election.cc:290] T 4f1c639a483f4827997eae6f3a6e375d P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.037789 23974 raft_consensus.cc:3060] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 11 FOLLOWER]: Advancing to term 12
I20251215 08:59:59.038497 21946 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:59.039608 21946 leader_election.cc:304] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e; no voters: beefe15185ac42ef9a20251007f39a56
I20251215 08:59:58.950587 24639 raft_consensus.cc:515] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:58.976533 24602 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.049937 21979 raft_consensus.cc:2468] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 9.
I20251215 08:59:58.959084 24512 raft_consensus.cc:515] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.064612 21414 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "beefe15185ac42ef9a20251007f39a56"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:59.104931 21414 raft_consensus.cc:606] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Received request to transfer leadership to TS beefe15185ac42ef9a20251007f39a56
I20251215 08:59:59.020552 24601 consensus_queue.cc:1243] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer 0a2fcb89589c401d8efc8d86da4ba0f4 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20251215 08:59:59.067296 21232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 215651us
I20251215 08:59:59.077131 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.078127 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.084362 24639 leader_election.cc:290] T 9d1675f3765f488e9d58a2ba2993457e P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.094726 24647 raft_consensus.cc:493] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f0824fa5450345719cac22fc6ae3ce2e)
I20251215 08:59:59.105384 21414 raft_consensus.cc:614] T caf0c0cd96174b4889e367b1f98d617e P 195a56ac72a84018880bfb3bf6e6ef1d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251215 08:59:59.066967 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.142501 21257 raft_consensus.cc:2468] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:59.142968 24371 consensus_queue.cc:1059] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251215 08:59:59.144003 24371 raft_consensus.cc:3055] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Stepping down as leader of term 8
I20251215 08:59:59.144065 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88823d4c370d4c6a96a6ba54418638eb" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:59.144615 24371 raft_consensus.cc:740] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:59.145581 21414 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f757e923d11a495091c94621efa6e4c1"
dest_uuid: "195a56ac72a84018880bfb3bf6e6ef1d"
mode: GRACEFUL
new_leader_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
from {username='slave'} at 127.0.0.1:34992
I20251215 08:59:59.146147 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84bdfb60f8fa4b39bc8cbd437a0809f2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.145681 24371 consensus_queue.cc:260] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:59.146565 21257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.147010 24371 raft_consensus.cc:3060] T 88823d4c370d4c6a96a6ba54418638eb P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Advancing to term 9
I20251215 08:59:59.146169 21414 raft_consensus.cc:606] T f757e923d11a495091c94621efa6e4c1 P 195a56ac72a84018880bfb3bf6e6ef1d [term 8 LEADER]: Received request to transfer leadership to TS 0a2fcb89589c401d8efc8d86da4ba0f4
I20251215 08:59:59.147512 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01ad5646d4d04a6a996ce407925b8650" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.148874 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.149849 21261 raft_consensus.cc:2393] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 7: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:59.104529 24636 log.cc:927] Time spent T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
W20251215 08:59:59.103685 24634 log.cc:927] Time spent T d20229ce83d84c8ba85a5fc4604d817d P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.332s user 0.001s sys 0.000s
I20251215 08:59:59.148833 20109 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251215 08:59:59.104882 24646 raft_consensus.cc:2804] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:59.153483 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.103893 24643 raft_consensus.cc:2764] T a4e724e065b349faafa05321d2582972 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20251215 08:59:59.172725 21264 raft_consensus.cc:2393] T 9d1675f3765f488e9d58a2ba2993457e P beefe15185ac42ef9a20251007f39a56 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d in current term 11: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:59.149495 24569 raft_consensus.cc:2749] T 641cd3a624114cf98d427a39378b6da6 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:59.154866 24248 consensus_queue.cc:1059] T 9582bb1875204089bca2e38a3ff426b3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.041s
W20251215 08:59:59.197109 21306 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251215 08:59:59.196662 21230 leader_election.cc:336] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.778s (SENT)
W20251215 08:59:59.185065 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 145428us
W20251215 08:59:59.200843 21944 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 170584us
I20251215 08:59:59.106799 24409 consensus_queue.cc:1059] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.068s
I20251215 08:59:59.105403 24629 consensus_queue.cc:1048] T e80fc34b30d346baa0dfec4df957cd9f P beefe15185ac42ef9a20251007f39a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.070s
I20251215 08:59:59.208709 21263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:59.209064 21230 leader_election.cc:336] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.817s (SENT)
I20251215 08:59:59.209520 21230 leader_election.cc:304] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:59.212738 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1936) took 2066 ms. Trace:
I20251215 08:59:59.213687 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b36e018bdd140648e80e809a47269c8" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.213164 21334 rpcz_store.cc:276] 1215 08:59:57.146494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.329405 (+1182911us) service_pool.cc:225] Handling call
1215 08:59:58.347161 (+ 17756us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:58.398584 (+ 51423us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:58.398613 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:58.414678 (+ 16065us) log.cc:844] Serialized 52 byte log entry
1215 08:59:58.550821 (+136143us) raft_consensus.cc:1645] Marking committed up to 9
1215 08:59:58.880662 (+329841us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.172737 (+292075us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.212404 (+ 39667us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.212477 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:59.212521 (+ 44us) raft_consensus.cc:1685] finished
1215 08:59:59.212537 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.212681 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17847,"prepare.queue_time_us":390826,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":4175,"raft.queue_time_us":129990,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":20523,"spinlock_wait_cycles":32211712,"thread_start_us":645769,"threads_started":4,"wal-append.queue_time_us":254702}
I20251215 08:59:59.200361 24562 leader_election.cc:290] T 109b80fba7b544e9946db3216c2c7581 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:59.058477 24625 consensus_queue.cc:237] T 676d761e0efc443cad981187af43b8d9 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.221223 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2014) took 2137 ms. Trace:
I20251215 08:59:59.221439 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2015) took 2114 ms. Trace:
I20251215 08:59:59.222175 21340 rpcz_store.cc:276] 1215 08:59:57.083634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.268797 (+1185163us) service_pool.cc:225] Handling call
1215 08:59:58.269160 (+ 363us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:58.342387 (+ 73227us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:58.342415 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:58.449486 (+107071us) log.cc:844] Serialized 26 byte log entry
1215 08:59:58.614117 (+164631us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:58.910249 (+296132us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.141596 (+231347us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.141678 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:59.220987 (+ 79309us) raft_consensus.cc:1685] finished
1215 08:59:59.221034 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.221163 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57682,"prepare.queue_time_us":373448,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":105291,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":152,"thread_start_us":618662,"threads_started":4,"wal-append.queue_time_us":255871}
I20251215 08:59:59.222617 21332 rpcz_store.cc:276] 1215 08:59:57.107211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.277936 (+1170725us) service_pool.cc:225] Handling call
1215 08:59:58.311809 (+ 33873us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:58.333491 (+ 21682us) raft_consensus.cc:1534] Early marking committed up to index 2
1215 08:59:58.333521 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:58.349827 (+ 16306us) log.cc:844] Serialized 26 byte log entry
1215 08:59:58.504717 (+154890us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:58.504758 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:58.504827 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:59.221240 (+716413us) raft_consensus.cc:1685] finished
1215 08:59:59.221266 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.221390 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2974,"raft.queue_time_us":27988,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":7290,"thread_start_us":79378,"threads_started":1,"wal-append.queue_time_us":192893}
I20251215 08:59:59.157578 24647 raft_consensus.cc:515] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.229605 21332 raft_consensus.cc:1275] T 458cec829de44303951b997be68accde P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 3 FOLLOWER]: Refusing update from remote peer beefe15185ac42ef9a20251007f39a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251215 08:59:59.230331 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1966) took 1406 ms. Trace:
I20251215 08:59:59.230721 21332 rpcz_store.cc:276] 1215 08:59:57.823422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.224213 (+1400791us) service_pool.cc:225] Handling call
1215 08:59:59.226126 (+ 1913us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.230054 (+ 3928us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.230270 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40576}
I20251215 08:59:59.243270 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.249792 21264 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 5.
I20251215 08:59:59.251273 24646 raft_consensus.cc:697] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Leader. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:59.234692 24621 raft_consensus.cc:2749] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:59.253773 24621 raft_consensus.cc:2749] T 9b36e018bdd140648e80e809a47269c8 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251215 08:59:59.253834 24646 consensus_queue.cc:237] T 4c50acc4d0854bd08c6afc283d478ca7 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.254590 21263 raft_consensus.cc:2468] T 5d93e1c6de78493c926c5be7096cfb31 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 9.
I20251215 08:59:59.255141 24659 raft_consensus.cc:493] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:59.255599 24659 raft_consensus.cc:515] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.258366 24659 leader_election.cc:290] T 080eec653ea04179a0dceafb6e3ffe47 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.259407 24651 consensus_queue.cc:1048] T 080eec653ea04179a0dceafb6e3ffe47 P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 08:59:59.241328 24657 raft_consensus.cc:493] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:59.261541 24602 leader_election.cc:290] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.261385 24657 raft_consensus.cc:515] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.263620 24657 leader_election.cc:290] T b3680af750ef47df88bf5be33eb18fc7 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.266314 24618 leader_election.cc:290] T 3b85e43f77684deab6a05166ac30cc14 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:59.267477 21260 raft_consensus.cc:2468] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 6.
I20251215 08:59:59.269256 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2023) took 1103 ms. Trace:
I20251215 08:59:59.269713 21332 rpcz_store.cc:276] 1215 08:59:58.165922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.231678 (+1065756us) service_pool.cc:225] Handling call
1215 08:59:59.268695 (+ 37017us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.268865 (+ 170us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:59.268888 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:59.268928 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:59.268972 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.269066 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.269192 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251215 08:59:59.240731 21945 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:59.274642 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7bfa9bddb4b48d996b8b347c898a3b7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:59.302599 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 353198us
I20251215 08:59:59.303666 21307 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:59.304631 24657 raft_consensus.cc:2749] T 60989b82e8354098aac2066e31697953 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:59.306000 21307 leader_election.cc:304] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:59.206434 24620 consensus_queue.cc:237] T 9582bb1875204089bca2e38a3ff426b3 P 195a56ac72a84018880bfb3bf6e6ef1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.252033 24489 leader_election.cc:290] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:59.310401 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfcf2c8cd22a488aa741f1cede6e86ef" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.267915 24482 leader_election.cc:290] T 01ad5646d4d04a6a996ce407925b8650 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
I20251215 08:59:59.266316 24647 leader_election.cc:290] T dd5a10a353d64f5c826c27e4a01f6132 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.313233 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" is_pre_election: true
I20251215 08:59:59.317203 21332 raft_consensus.cc:2410] T 3b85e43f77684deab6a05166ac30cc14 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 195a56ac72a84018880bfb3bf6e6ef1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251215 08:59:59.306923 24657 raft_consensus.cc:2749] T 64b5a2ef4843492085aca418ad649914 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251215 08:59:59.314162 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 105153us
W20251215 08:59:59.318567 21230 leader_election.cc:336] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.975s (SENT)
I20251215 08:59:59.318945 21230 leader_election.cc:304] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:59.321714 21946 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 117005us
I20251215 08:59:59.324613 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:59.326457 21230 leader_election.cc:336] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.045s (SENT)
I20251215 08:59:59.326862 21230 leader_election.cc:304] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:59.331562 21230 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:59.328682 21264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba393161b7454821b7584a44b64bf6b2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.332563 21258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d93e1c6de78493c926c5be7096cfb31" candidate_uuid: "f0824fa5450345719cac22fc6ae3ce2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.329217 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3680af750ef47df88bf5be33eb18fc7" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:59.334342 21332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4"
I20251215 08:59:59.329566 21229 leader_election.cc:304] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:59.335129 21332 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:59.307325 23974 raft_consensus.cc:515] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.344412 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59576dccc86c4396bdbb3706a9750747" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:59.345170 21259 raft_consensus.cc:3060] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:59.341655 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080eec653ea04179a0dceafb6e3ffe47" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:59.342485 21257 raft_consensus.cc:1240] T 4f1c639a483f4827997eae6f3a6e375d P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:59.327796 24650 raft_consensus.cc:493] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 08:59:59.328505 24647 raft_consensus.cc:2749] T d69415d618114174b070bbc19a876119 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:59.361842 21307 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:59.328392 24512 leader_election.cc:290] T 7edcdb5749ae4e59ab17648fe97390e2 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.363516 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd5a10a353d64f5c826c27e4a01f6132" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:59.351495 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "beefe15185ac42ef9a20251007f39a56" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:59.352543 21230 leader_election.cc:336] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.511s (SENT)
W20251215 08:59:59.353605 21305 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:59.333406 24482 raft_consensus.cc:2749] T 6053bc9278744c6b93f60f788aa7b06e P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:59.339038 24409 raft_consensus.cc:3055] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Stepping down as leader of term 9
W20251215 08:59:59.363936 21231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:59.364601 21979 raft_consensus.cc:3060] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Advancing to term 8
I20251215 08:59:59.366899 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d20229ce83d84c8ba85a5fc4604d817d" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.341363 24602 raft_consensus.cc:2804] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 9 FOLLOWER]: Leader election won for term 9
I20251215 08:59:59.376127 24626 raft_consensus.cc:515] T c61929339a494e3fb5f327784d94d817 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
W20251215 08:59:59.376835 24637 log.cc:927] Time spent T b7bfa9bddb4b48d996b8b347c898a3b7 P 0a2fcb89589c401d8efc8d86da4ba0f4: Append to log took a long time: real 0.493s user 0.000s sys 0.001s
I20251215 08:59:59.380556 21946 leader_election.cc:304] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, beefe15185ac42ef9a20251007f39a56
I20251215 08:59:59.364562 21230 leader_election.cc:304] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:59.396608 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 44128.7us
W20251215 08:59:59.397727 21230 leader_election.cc:336] T 458cec829de44303951b997be68accde P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.852s (SENT)
I20251215 08:59:59.381505 21259 raft_consensus.cc:2410] T 59576dccc86c4396bdbb3706a9750747 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251215 08:59:59.399607 21332 raft_consensus.cc:2468] T 3e9ae02f9a5345069e7afbf6d52e9299 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:59.399619 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e80fc34b30d346baa0dfec4df957cd9f" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.393651 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0922c97d6dce49f0a3b98d200f4f651e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.405646 21946 leader_election.cc:304] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters: 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:59.409783 21230 leader_election.cc:336] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.867s (SENT)
I20251215 08:59:59.410317 21230 leader_election.cc:304] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:59.411830 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 62428.1us
I20251215 08:59:59.411942 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7edcdb5749ae4e59ab17648fe97390e2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:59.413367 21979 raft_consensus.cc:2468] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beefe15185ac42ef9a20251007f39a56 in term 8.
I20251215 08:59:59.458191 21259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4ed801d85bc47d7ac87033f0799bad2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
W20251215 08:59:59.459347 21944 leader_election.cc:336] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 1.738s (SENT)
I20251215 08:59:59.459797 21944 leader_election.cc:304] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0824fa5450345719cac22fc6ae3ce2e; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
W20251215 08:59:59.413848 21230 leader_election.cc:336] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.047s (SENT)
I20251215 08:59:59.460458 21230 leader_election.cc:304] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:59.466498 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07b6839b646d44fbac5ae9e9e2a65daa" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.382691 24409 raft_consensus.cc:740] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f0824fa5450345719cac22fc6ae3ce2e, State: Running, Role: LEADER
I20251215 08:59:59.467813 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6e3fc5ec2cc402a96b9fcf2fa3cc5c3" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:59.403638 24659 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:59.403251 24512 raft_consensus.cc:2749] T d8b3d79d7a014e568e385110044efb63 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251215 08:59:59.418224 24594 raft_consensus.cc:2749] T 79790fd760ff4ae1b3879326decc531a P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:59.409205 24602 raft_consensus.cc:697] T a4e724e065b349faafa05321d2582972 P beefe15185ac42ef9a20251007f39a56 [term 9 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
W20251215 08:59:59.464365 24652 log.cc:927] Time spent T e80fc34b30d346baa0dfec4df957cd9f P 195a56ac72a84018880bfb3bf6e6ef1d: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20251215 08:59:59.475795 21229 leader_election.cc:304] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beefe15185ac42ef9a20251007f39a56, f0824fa5450345719cac22fc6ae3ce2e; no voters:
I20251215 08:59:59.476511 21979 raft_consensus.cc:2468] T a6e3fc5ec2cc402a96b9fcf2fa3cc5c3 P f0824fa5450345719cac22fc6ae3ce2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 7.
I20251215 08:59:59.479564 21379 leader_election.cc:304] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d; no voters:
W20251215 08:59:59.480871 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 68358.1us
I20251215 08:59:59.418200 24482 raft_consensus.cc:2749] T ba393161b7454821b7584a44b64bf6b2 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251215 08:59:59.380142 24633 raft_consensus.cc:515] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:59.465934 24662 raft_consensus.cc:2804] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251215 08:59:59.486227 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676d761e0efc443cad981187af43b8d9" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:59.503607 21230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.66:36421, user_credentials={real_user=slave}} blocked reactor thread for 89771.7us
W20251215 08:59:59.504557 21229 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251215 08:59:59.505460 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d471c9f4eb74ea3a20cec86631e4ab0" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:59.379429 24650 raft_consensus.cc:515] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.512456 21975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1c639a483f4827997eae6f3a6e375d" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
W20251215 08:59:59.508036 21230 leader_election.cc:336] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.049s (SENT)
I20251215 08:59:59.513579 21230 leader_election.cc:304] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:59.514422 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1675f3765f488e9d58a2ba2993457e" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:59.475445 24670 raft_consensus.cc:2749] T ac8397b87ffc453b8f468eee77a8e410 P f0824fa5450345719cac22fc6ae3ce2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251215 08:59:59.519850 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "458cec829de44303951b997be68accde" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:59.515112 21980 raft_consensus.cc:2468] T 9d1675f3765f488e9d58a2ba2993457e P f0824fa5450345719cac22fc6ae3ce2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 11.
I20251215 08:59:59.517725 21259 raft_consensus.cc:2393] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 7: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
W20251215 08:59:59.506713 24333 consensus_meta.cc:287] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.504s user 0.006s sys 0.005s
I20251215 08:59:59.492777 24666 raft_consensus.cc:493] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:59.510479 24662 raft_consensus.cc:493] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 08:59:59.468423 24673 consensus_queue.cc:1059] T 4f1c639a483f4827997eae6f3a6e375d P f0824fa5450345719cac22fc6ae3ce2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20251215 08:59:59.509723 24621 raft_consensus.cc:2804] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader election won for term 8
I20251215 08:59:59.528919 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2024) took 1352 ms. Trace:
I20251215 08:59:59.532891 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2016) took 2410 ms. Trace:
I20251215 08:59:59.534931 21332 rpcz_store.cc:276] 1215 08:59:58.176754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.400652 (+1223898us) service_pool.cc:225] Handling call
1215 08:59:59.401069 (+ 417us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.401313 (+ 244us) raft_consensus.cc:1534] Early marking committed up to index 8
1215 08:59:59.459346 (+ 58033us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.528311 (+ 68965us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.528501 (+ 190us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:59.528559 (+ 58us) raft_consensus.cc:1645] Marking committed up to 8
1215 08:59:59.528621 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.528719 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.528860 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17038,"prepare.queue_time_us":76843,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":6852,"raft.queue_time_us":1220,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":5650,"thread_start_us":51059,"threads_started":1}
I20251215 08:59:59.506220 24667 raft_consensus.cc:2749] T b7bfa9bddb4b48d996b8b347c898a3b7 P beefe15185ac42ef9a20251007f39a56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251215 08:59:59.539076 21262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf0c0cd96174b4889e367b1f98d617e" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:59.541502 21407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32818 (request call id 1916) took 1194 ms. Trace:
I20251215 08:59:59.532380 24662 raft_consensus.cc:3060] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 10 FOLLOWER]: Advancing to term 11
I20251215 08:59:59.541392 24621 raft_consensus.cc:697] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [term 8 LEADER]: Becoming Leader. State: Replica: beefe15185ac42ef9a20251007f39a56, State: Running, Role: LEADER
I20251215 08:59:59.549831 24621 consensus_queue.cc:237] T c61929339a494e3fb5f327784d94d817 P beefe15185ac42ef9a20251007f39a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.550590 21307 leader_election.cc:304] T f800ccb361734ac18102a3502ee639d8 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:59.552464 24650 leader_election.cc:290] T 4c50acc4d0854bd08c6afc283d478ca7 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.552855 21409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59102 (request call id 352) took 2568 ms. Trace:
I20251215 08:59:59.529521 24562 raft_consensus.cc:493] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 195a56ac72a84018880bfb3bf6e6ef1d)
I20251215 08:59:59.553848 24633 leader_election.cc:290] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), 195a56ac72a84018880bfb3bf6e6ef1d (127.19.163.67:37447)
W20251215 08:59:59.554260 21341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49734 (request call id 2020) took 2352 ms (client timeout 3112 ms). Trace:
I20251215 08:59:59.529320 24333 raft_consensus.cc:515] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
W20251215 08:59:59.554678 21341 rpcz_store.cc:269] 1215 08:59:57.202280 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:57.434792 (+232512us) service_pool.cc:225] Handling call
1215 08:59:59.554083 (+2119291us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001a9fd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:59.554177 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":240928128}
I20251215 08:59:59.556242 24333 leader_election.cc:290] T 88823d4c370d4c6a96a6ba54418638eb P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 10 election: Requested vote from peers beefe15185ac42ef9a20251007f39a56 (127.19.163.65:46193), 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421)
I20251215 08:59:59.556309 21341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2026) took 1353 ms. Trace:
W20251215 08:59:59.557443 24668 log.cc:927] Time spent T 2c10c3de775846868774c98cc0a75094 P beefe15185ac42ef9a20251007f39a56: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20251215 08:59:59.534982 24274 raft_consensus.cc:2749] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251215 08:59:59.559005 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6fb62879dd443a18eb94c506c7a82a2" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.559584 21260 raft_consensus.cc:2468] T c6fb62879dd443a18eb94c506c7a82a2 P beefe15185ac42ef9a20251007f39a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in term 8.
W20251215 08:59:59.560290 24649 consensus_meta.cc:287] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.539s user 0.010s sys 0.000s
I20251215 08:59:59.560328 21260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1db12563817b44429a055b7dcdd066c3" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:59.538210 24680 raft_consensus.cc:2804] T 3e9ae02f9a5345069e7afbf6d52e9299 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Leader election won for term 7
I20251215 08:59:59.560824 24649 raft_consensus.cc:515] T 59576dccc86c4396bdbb3706a9750747 P 195a56ac72a84018880bfb3bf6e6ef1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } }
I20251215 08:59:59.561748 23974 leader_election.cc:290] T 676d761e0efc443cad981187af43b8d9 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 12 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
W20251215 08:59:59.562700 23959 consensus_meta.cc:287] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d: Time spent flushing consensus metadata: real 0.663s user 0.010s sys 0.000s
I20251215 08:59:59.563076 23959 raft_consensus.cc:515] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.563313 24662 raft_consensus.cc:515] T 3cf7b0165c00477b9c1b94c6a89052a6 P f0824fa5450345719cac22fc6ae3ce2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.564190 21413 raft_consensus.cc:1240] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [term 7 FOLLOWER]: Rejecting Update request from peer f0824fa5450345719cac22fc6ae3ce2e for earlier term 6. Current term is 7. Ops: []
I20251215 08:59:59.533653 24666 raft_consensus.cc:515] T 8d7af00ee6834714a059e045b3c13322 P f0824fa5450345719cac22fc6ae3ce2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.529183 24409 consensus_queue.cc:260] T 1db12563817b44429a055b7dcdd066c3 P f0824fa5450345719cac22fc6ae3ce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } } peers { permanent_uuid: "f0824fa5450345719cac22fc6ae3ce2e" member_type: VOTER last_known_addr { host: "127.19.163.68" port: 45831 } attrs { promote: false } }
I20251215 08:59:59.564963 23959 leader_election.cc:290] T f800ccb361734ac18102a3502ee639d8 P 195a56ac72a84018880bfb3bf6e6ef1d [CANDIDATE]: Term 7 election: Requested vote from peers 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421), f0824fa5450345719cac22fc6ae3ce2e (127.19.163.68:45831)
I20251215 08:59:59.541870 21407 rpcz_store.cc:276] 1215 08:59:58.344053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.344222 (+ 169us) service_pool.cc:225] Handling call
1215 08:59:58.344627 (+ 405us) raft_consensus.cc:1471] Updating replica for 2 ops
1215 08:59:58.839856 (+495229us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:58.839903 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1215 08:59:58.865945 (+ 26042us) log.cc:844] Serialized 52 byte log entry
1215 08:59:59.010611 (+144666us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:59.218082 (+207471us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.225681 (+ 7599us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.225777 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:59.367522 (+141745us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b18003a8d60. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f398d957f21 00007f398d964901 00007f3984b05fe3 00007f3984b083b1 00007f3984b08369 00007f3984b082f9 00007f3984b082c1 00007f3984b075bd 0000000000379254 0000000000379099 00007f3984ae93aa
1215 08:59:59.527458 (+159936us) raft_consensus.cc:1685] finished
1215 08:59:59.538050 (+ 10592us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.538195 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22142,"prepare.queue_time_us":153252,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":6540,"raft.queue_time_us":114818,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":39785,"spinlock_wait_cycles":7402240,"thread_start_us":670560,"threads_started":3,"wal-append.queue_time_us":282539}
W20251215 08:59:59.565090 21230 leader_election.cc:336] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.083s (SENT)
I20251215 08:59:59.574978 21337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49750 (request call id 1943) took 2426 ms. Trace:
I20251215 08:59:59.575491 21230 leader_election.cc:304] T 124c9efc80174ca585aaee7e0f8a12f0 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, f0824fa5450345719cac22fc6ae3ce2e
W20251215 08:59:59.552459 21335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49750 (request call id 1956) took 1832 ms (client timeout 2083 ms). Trace:
I20251215 08:59:59.576344 21979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c61929339a494e3fb5f327784d94d817" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e"
W20251215 08:59:59.576694 21230 leader_election.cc:336] T f4ed801d85bc47d7ac87033f0799bad2 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.173s (SENT)
I20251215 08:59:59.577165 21979 raft_consensus.cc:2393] T c61929339a494e3fb5f327784d94d817 P f0824fa5450345719cac22fc6ae3ce2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a2fcb89589c401d8efc8d86da4ba0f4 in current term 8: Already voted for candidate beefe15185ac42ef9a20251007f39a56 in this term.
I20251215 08:59:59.535476 21339 rpcz_store.cc:276] 1215 08:59:57.122274 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.294802 (+1172528us) service_pool.cc:225] Handling call
1215 08:59:58.310425 (+ 15623us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:58.310875 (+ 450us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:58.310917 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:58.352205 (+ 41288us) log.cc:844] Serialized 26 byte log entry
1215 08:59:58.597225 (+245020us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:58.894059 (+296834us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.066906 (+172847us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.066984 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:59.376321 (+309337us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b440002304c. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3984946781 00007f3984a14ed5 00007f3984a158f2 00007f398775f57b 00007f398775ae4f 00007f398774ec0e 00007f398774df56 00007f398774db78 00007f39877575c1 00007f3987757579 00007f3987757509
1215 08:59:59.525604 (+149283us) raft_consensus.cc:1685] finished
1215 08:59:59.532716 (+ 7112us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.532838 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7499,"prepare.queue_time_us":364830,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":4880,"raft.queue_time_us":101566,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":7007,"spinlock_wait_cycles":19894656,"thread_start_us":549945,"threads_started":3,"wal-append.queue_time_us":280998}
W20251215 08:59:59.578552 21230 leader_election.cc:336] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.328s (SENT)
I20251215 08:59:59.578948 21230 leader_election.cc:304] T 07b6839b646d44fbac5ae9e9e2a65daa P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:59.579056 21339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2027) took 1362 ms. Trace:
I20251215 08:59:59.579216 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2021) took 2351 ms. Trace:
I20251215 08:59:59.579437 21339 rpcz_store.cc:276] 1215 08:59:58.216689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.578298 (+1361609us) service_pool.cc:225] Handling call
1215 08:59:59.578589 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.578758 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:59.578782 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:59.578831 (+ 49us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:59.578861 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.578908 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.579009 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:59.579558 21340 rpcz_store.cc:276] 1215 08:59:57.228102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.224176 (+1996074us) service_pool.cc:225] Handling call
1215 08:59:59.224573 (+ 397us) raft_consensus.cc:1471] Updating replica for 1 ops
1215 08:59:59.234048 (+ 9475us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:59.234082 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1215 08:59:59.242364 (+ 8282us) log.cc:844] Serialized 26 byte log entry
1215 08:59:59.495464 (+253100us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:59.525861 (+ 30397us) log.cc:844] Serialized 14 byte log entry
1215 08:59:59.527853 (+ 1992us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.527927 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1215 08:59:59.579036 (+ 51109us) raft_consensus.cc:1685] finished
1215 08:59:59.579058 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.579168 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10846,"prepare.queue_time_us":32235,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":29,"raft.queue_time_us":41594,"raft.run_cpu_time_us":40,"raft.run_wall_time_us":40,"spinlock_wait_cycles":10618624,"thread_start_us":187740,"threads_started":1,"wal-append.queue_time_us":304333}
I20251215 08:59:59.566434 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c10c3de775846868774c98cc0a75094" candidate_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
I20251215 08:59:59.582214 21340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2031) took 1345 ms. Trace:
I20251215 08:59:59.553978 24562 raft_consensus.cc:515] T a76bbffe260349a49c7cad7a3de2a9da P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beefe15185ac42ef9a20251007f39a56" member_type: VOTER last_known_addr { host: "127.19.163.65" port: 46193 } } peers { permanent_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" member_type: VOTER last_known_addr { host: "127.19.163.67" port: 37447 } } peers { permanent_uuid: "0a2fcb89589c401d8efc8d86da4ba0f4" member_type: VOTER last_known_addr { host: "127.19.163.66" port: 36421 } }
I20251215 08:59:59.582556 21340 rpcz_store.cc:276] 1215 08:59:58.237323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.581478 (+1344155us) service_pool.cc:225] Handling call
1215 08:59:59.581737 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.581905 (+ 168us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:59.581927 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:59.581967 (+ 40us) raft_consensus.cc:1645] Marking committed up to 4
1215 08:59:59.582014 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.582065 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.582170 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:59.575515 21337 rpcz_store.cc:276] 1215 08:59:57.148471 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.527478 (+1379007us) service_pool.cc:225] Handling call
1215 08:59:58.552705 (+ 25227us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:58.749817 (+197112us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.574753 (+824936us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400279cd4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb4a81 00007f3987bb34be 00007f398cb67f49 00007f39860f4d34 00007f39860f4cc2 00007f39860f4bf1 00007f39860f4b6c 00007f39860f3e22 00007f3985ef355c 00007f3985ef2996
1215 08:59:59.574918 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":272656128}
W20251215 08:59:59.576418 21335 rpcz_store.cc:269] 1215 08:59:57.720028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:58.231208 (+511180us) service_pool.cc:225] Handling call
1215 08:59:59.552274 (+1321066us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54000d20d4. stack: 00007f3984ad4083 00007f3984ad3f80 00007f3984426574 00007f398e0e16b4 00007f398e0e1649 00007f3987b1d5a1 00007f3987bb7f7a 00007f398cb69647 00007f39860f9834 00007f39860f97c2 00007f39860f96f1 00007f39860f966c 00007f39860f8922 00007f3985ef355c 00007f3985ef2996 00007f3985ef233f
1215 08:59:59.552405 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":670970624}
I20251215 08:59:59.556735 21341 rpcz_store.cc:276] 1215 08:59:58.203440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.555558 (+1352118us) service_pool.cc:225] Handling call
1215 08:59:59.555857 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.556033 (+ 176us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:59.556056 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:59.556094 (+ 38us) raft_consensus.cc:1645] Marking committed up to 5
1215 08:59:59.556125 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.556175 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.556261 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:59.568552 21980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c50acc4d0854bd08c6afc283d478ca7" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f0824fa5450345719cac22fc6ae3ce2e" is_pre_election: true
I20251215 08:59:59.575577 21262 raft_consensus.cc:3060] T caf0c0cd96174b4889e367b1f98d617e P beefe15185ac42ef9a20251007f39a56 [term 3 FOLLOWER]: Advancing to term 4
I20251215 08:59:59.585400 21266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b85e43f77684deab6a05166ac30cc14" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56" is_pre_election: true
W20251215 08:59:59.579911 21230 leader_election.cc:336] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0a2fcb89589c401d8efc8d86da4ba0f4 (127.19.163.66:36421): Timed out: RequestConsensusVote RPC to 127.19.163.66:36421 timed out after 2.405s (SENT)
I20251215 08:59:59.595952 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2034) took 1351 ms. Trace:
I20251215 08:59:59.596401 21266 raft_consensus.cc:2468] T 3b85e43f77684deab6a05166ac30cc14 P beefe15185ac42ef9a20251007f39a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195a56ac72a84018880bfb3bf6e6ef1d in term 5.
I20251215 08:59:59.596381 21230 leader_election.cc:304] T 60989b82e8354098aac2066e31697953 P beefe15185ac42ef9a20251007f39a56 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beefe15185ac42ef9a20251007f39a56; no voters: 0a2fcb89589c401d8efc8d86da4ba0f4, 195a56ac72a84018880bfb3bf6e6ef1d
I20251215 08:59:59.596428 21335 rpcz_store.cc:276] 1215 08:59:58.244714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.595186 (+1350472us) service_pool.cc:225] Handling call
1215 08:59:59.595450 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.595615 (+ 165us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:59.595643 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:59.595679 (+ 36us) raft_consensus.cc:1645] Marking committed up to 3
1215 08:59:59.595706 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.595766 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.595888 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:59.599325 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49734 (request call id 2036) took 1349 ms. Trace:
I20251215 08:59:59.599915 21335 rpcz_store.cc:276] 1215 08:59:58.250029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1215 08:59:59.598439 (+1348410us) service_pool.cc:225] Handling call
1215 08:59:59.598765 (+ 326us) raft_consensus.cc:1471] Updating replica for 0 ops
1215 08:59:59.598952 (+ 187us) raft_consensus.cc:1534] Early marking committed up to index 0
1215 08:59:59.598998 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1215 08:59:59.599048 (+ 50us) raft_consensus.cc:1645] Marking committed up to 2
1215 08:59:59.599088 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1215 08:59:59.599151 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1215 08:59:59.599253 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251215 08:59:59.602387 21335 raft_consensus.cc:3060] T 2c10c3de775846868774c98cc0a75094 P 0a2fcb89589c401d8efc8d86da4ba0f4 [term 7 FOLLOWER]: Advancing to term 8
W20251215 08:59:59.607441 21307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.163.67:37447, user_credentials={real_user=slave}} blocked reactor thread for 56922.5us
I20251215 08:59:59.608100 21265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e9ae02f9a5345069e7afbf6d52e9299" candidate_uuid: "195a56ac72a84018880bfb3bf6e6ef1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "beefe15185ac42ef9a20251007f39a56"
I20251215 08:59:59.608390 21307 leader_election.cc:304] T 4d1d9a9be078441587cf5da752d59c33 P 0a2fcb89589c401d8efc8d86da4ba0f4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a2fcb89589c401d8efc8d86da4ba0f4; no voters: 195a56ac72a84018880bfb3bf6e6ef1d, f0824fa5450345719cac22fc6ae3ce2e
I20251215 08:59:59.608819 21265 raft_consensus.cc:3060] T 3e9ae02f9a5345069e7afbf6d52e9299 P beefe15185ac42ef9a20251007f39a56 [term 6 FOLLOWER]: Advancing to term 7
I20251215 08:59:59.567327 21261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "